Analyzing Automation Results with Nikita Sidorevich

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Nikita all about analyzing automation results. The kid is an engineer with a passion for web development, design, best practices and software development methodologies. He's also currently the product manager at ZBrunner, which is a tool that helps with smart analysis of test results. I think it's a highly overlooked area with automation is reporting and that's the type of things we'll be diving in today. So if you want to learn how to shorten your release cycle and deliver what high quality, you don't want to miss this episode, check it out. The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Thanks. Cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Nikita. Welcome to the Guild.

Nigel. Hi everyone.

Awesome, awesome to have you. Really excited to have you on the show today. Nikita, before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

Yeah, yeah, sure. It's probably a good idea to talk a little bit about myself. Yeah, my name is Nikita and I started my journey in this industry as a software developer. And in fact this year marks my 10 year anniversary in the industry. And well, the thing is that I started to work for a huge service provider company and for like seven years I used to work as a contractor and I used to work for mostly enterprise scale organizations and well, I guess it would be fair to say that during that period of time I realized how messy automation can be. Right. So anyways, let's keep this for later. So except for the development, I used to do a little bit of DevOps desktops. Later I started to manage teams and projects and yeah, I mean like three years ago I joined ZipRunner team. And in fact back in the days it wasn't really called ZipRunner, so it was just an internal tool we were developing for our own needs. And yeah, a lot of things happened since then. But now, well, three years later, here I am playing the role of zbrunner Product manager.

Very cool. So you mentioned Azure. You've been a developer for 10 years. Automation could become messy. What are some of the Reasons for that.

I guess maybe 10 years ago, automation really became trendy, like a trendy thing. And the higher the complexity of the application you're developing or of your business like, the more test cases have to be automated. And that's especially critical for some domains. For example, for financial sector, pretty much everything needs to be automated. And the problem is that, well, automation by definition was aimed to ease things, to simplify things for you, so it can speed up processes that were done manually before. And I believe the biggest problem with automation, when it becomes less effective than it should be. And this effectiveness is especially important because I've seen situations when testing becomes a bottleneck and test maintenance becomes a nightmare, when you have to deal with a lot of unstable tests, when results are not really reliable. And that's when you have to think about efficiency of your automation and maybe you're doing things wrong if you got in such a situation.

Absolutely. And I think the efficiency part is a big issue. A lot of times it's great to create automated tests. Right. But then what do you do after the tests are written and you have to run them in parallel? You have to run them continuously and they're failing and they're flaky.

Yeah, correct.

I found working for Enterprise, most of my time was spent actually debugging failures and most of them weren't even failures. So I think that's where reporting comes in. Any thoughts around that?

Yeah. So in fact, yeah, I have a very similar feeling, you know, because I remember my days working, you know, like as part of large teams, and we had this, it was a set tradition, I would say. So every morning we started with test run results review and it could have taken for like up to two hours. And we even had a dedicated person responsible for that. It was called K Captain. So K Captain was coming, starting his workday from opening the Jenkins report, for example, and he was going through all of those automated test results and it wasn't always clear what went wrong, if it was like a latency issue or it was indeed an application bug. So a lot of debugging took place and sometimes we had to work as a whole team, like both test automation engineers, developers, just to make some sense out of those reports. So we're literally wasting hours of our time just doing that.

100%. I love that you're a developer, so I'm just curious to get your take on this. With automation, there's been this push towards AI machine learning, but I think a lot of it's been marketing focused. But you're an actual developer. How much do you think AI can actually help with automation?

I guess. Well, thing is that AI is a very broad term. Right. And I would say as a developer myself, you know, who was taking part in a lot of development conferences, I try to be proactive and I try to read a lot of stuff on the Internet. So it seems like nowadays AI is, well, it's turning into a buzzword and it's actually not, it's not like it happened yesterday. Right. So it's been like this for maybe last few years for sure. I believe that indeed there are aspects of AI and machine learning that can indeed help engineers to take care test automation engineers to take care of their issues with test automation. So for example, we have reappearing issues and sometimes you have to process too much information way more than you can keep in your head. And that's when machine learning can come handy. So basically some of the aspects of the analysis can be delegated to machine learning. And well, the best thing that you can train, you can train those machine learning algorithms, you can train those networks. And yeah, at some point of time you can really see the benefits of incorporating machine learning, especially in the aspects of test run results review. So that's what I think on that.

Yeah, no, I totally agree. I think the thing that people miss out on is machine learning is probably good for analyzing lots of data over history or over time and things like performance logs or even automation logs or test runs seems to be a perfect user case for machine learning. Is that true?

Correct? Yeah, exactly.

Cool. So how then could machine learning help with analyzing automated test results? What are some key areas you think it could actually benefit people from using something that has machine learning built into it to help you with your reporting test results analysis?

Yeah. So if we'll think about a routine that test automation engineers, you know, comes through. So we work with a lot of artifacts, most of those artifacts are test based. So now we have stuff like videos and screenshots, but we still have to analyze a lot of text. And even when it comes to images, we still can benefit from machine learning. So we can do visual regression testing. We can do that automatically with the help of systems built on top of AI. And when it comes to text analysis, we actually can do a lot of interesting stuff by, you know, well, in area of failure classification because like, well, failure is a very broad term once again, so you have to be very sure about what is causing this failure. So you have to do some sort of classification first. Well, without this classification, it's really Hard to understand what your reaction should be because if it's for example, an application issue, you should probably file a bug and go to developer and say, hey, this is something that needs to be fixed. If it's, for example, test needs to be updated once again, you want to have some guidance telling you that. So that's, I would say the aspect of classification is when it comes really helpful. All of those artifacts that we use to process manually, application logs, test logs, stack traces, responses from backend, that kind of stuff, all of this can be really processed by machine learning systems and it can help us to classify those issues. And by having this classification in place, we can just have an idea of what needs to be done about those issues so we can actually focus on what matters instead of spending our time analyzing those results ourselves manually.

Absolutely. And I love like this helps when you come in the morning or when the test results are done, you can look at it and you know which is most likely a real issue as opposed to maybe a flaky test or a known issue. That's what it sounds like to me.

Yeah.

So I guess along those lines then is once again, when I was working for a large enterprise, there'd be a bug introduced during a sprint and people knew about that. But we still would run our tests after every check in. But I still have to remember, oh, this, this is when a test failed, I'd have to say, oh, that's, this is actually not a real issue. It's a, it's a known issue, they're fixing it so I don't have to worry about it right now. But then I'd have to tag it in the code and then I forget to untag it and became a nightmare. So can machine learning or can reporting help filter these out, categorize these out as well?

Yeah, absolutely. So basically even in the way you described it. So it's basically a set of repeatable steps you had to do every day. So this is obviously something that can be done automatically and a lot of reporting tools are actually doing that. Well, I don't think that this particular, well, use case is a concern for machine learning. I mean machine learning can be incorporated if we're doing some probability algorithm, if we're writing some probability based algorithm. So machine learning can be used in this scenario. But yeah, most of those steps, for example, let's say that you have a test that failed yesterday and someone went ahead and filed a bug in project management tool, for example, or a bug tracker and it was linked, for example, it was linked to this failed test execution yesterday. So the next day it feels really stupid to do the same thing if it wasn't fixed, if this bug wasn't really addressed, if the issue wasn't fixed, it feels really stupid to do the same routine once again. So this is something that reporting tools help you to speed up to take off your. Yeah, work disk, probably.

Right. Yeah. And I guess also maybe not machine learning specific, but historically having that data, knowing that maybe this test failed two months ago, they fixed it and then it failed again and you don't remember what they did to fix it, you'd be able to go into the report and be able to kind of find that information, I would think, if it was set up correctly.

Yeah. So this correlation is especially important indeed. So when you have. Yeah, if you find that, basically if you see the failing test that has the same root cause, like the one that was failing for example, six months ago, and maybe there was a bug six months ago filed in a bug tracking system and someone fixed that bug, it would be really easy for you by having this correlation in place, it would be really easy for you to find what actually fixed that bug and speed up not only the, well, the failure analysis process, but the process of fixing the issue itself.

Absolutely. I guess once again, going back to my work experience, after every sprint, we'd have to have a review with our management over the automation results and what happened and we'd have to look at the trends and I'd have to keep. Keep track of everything in Excel spreadsheet to try to explain why test runs were less than they were before or what the failure rate, why it failed on one day but not the other. I guess also a lot of people don't leverage reporting to have that type of information or be able to maybe give up a dashboard for management as opposed to technical people. Is that something you see as a need as well for reporting?

Yeah, that is true. I mean, I would say that that actually depends on the business, on the organization. So from my experience, smaller businesses, smaller organizations, startups especially, they don't really pay much attention to the analysis process. And I believe one of the reasons behind that is the fact that they don't really do much automation as a small organization. As a startup, they're focused more on a delivery, sometimes forgetting about automation and quality, which is unfortunate. But once you have a running business, once you have complex software solution running and making you money, you are more and more concerned about automation and you definitely want to have more transparency and visibility over what's going on. So in our experience we've been talking a lot to people from the community, to representatives of organizations large and small. So we believe that it is especially important once organization reach certain scale, especially when it comes to enterprise scale organization that have, well, thousands of tests running. And when you have thousands of tests running, even if you don't have, well, you know, that big percent of the failures. For example, if you have like a 5% of tests failing every day, which is terrible if you ask me. But once again, if you think about that 5% out of, well, let's say 10,000 tests, the reviewing part can be a nightmare. But not only the reviewing part is problematic, but you want to understand what is actually going on. You want to know what tests are unstable, what tests are flaky. And yeah, so basically you can have an idea of what needs to be done, how those issues can be addressed, and maybe you should refactor some of your tests. So ideally you want your regression reports to be green always. And yeah, so in our experience that mostly comes at a certain level of organization maturity. So that's not really well applicable for an organization out there.

No, no, absolutely. And like you said, when you have large test suites that run for enterprise, even a small failure rate like 3%, that's a lot of tests to have to debug and maintain. And a lot of times I saw teams struggling to do it in time before the next test run. So eventually the sprint teams would be like, you know, they didn't know what was going on so they just would ignore the results. Seems like reporting could help. You say to each Sprint team, look at this dashboard for the sprint team. These are actual failures. There aren't known issues. They're not flaky tasks. You should look at these right away. I would think that would help get teams more involved as well and not just ignore the result.

That's a very good point. That actually makes me thinking about another thing is that sometimes some organizations do automation like this. So basically there is a development team that does a feature development and then there is another team and this team can be located in another country or another continent maybe and that is the team that is doing automation and it's really out of sync with the development. And yeah, so like you said, it doesn't really motivate developer to care about automation much because they're like, okay, that's this guy's problem. So once those things start running out of things, out of sync, that's when it became especially problematic. Once again, we are going Back to the thought that automation should be effective and in a scenario like this, well, the effectiveness of automation is probably, well, not very high, unfortunately.

Absolutely. So at the very beginning of the episode you talked about how you work at Zebron, but it wasn't Zebron when you started it. Just you were working on something and eventually whatever you're working on turned into the product. Zebrunter, we never actually talked about what is Zebron at a high level. So what is Zebrunter, I guess is the main question.

Yeah, that's a very good one. So yeah, As I said, ZipRunner initially started as an internal tool. Well, and it was basically an internal tool for a company that was in fact offering services and quality assurance. So basically it was a company that was a service provider. And our goal back in the days was to make test automation process as transparent for our customers as possible. And that's how ZipRunner was born. So it wasn't ZipRunner back in the days, but that's how this solution, the idea was born. And it all started from a set of relatively simple static reports and a very basic dashboard providing very high level idea like how many failing tests you have in a test run or for example, what is a daily pass rate, that kind of stuff. Well, pretty high level. And the thing is that while this organization was onboarding more and more customers, this tool came really handy to our teams. So at some point of time there was a dedicated team of a few developers who were doing this halftime and they started to evolve this tool. This unnamed unknown tool. Right. And around the same time we made it open source. And yeah, so we started to evolve it. We focused on engineering productivity and advanced analytics, more advanced than what I just mentioned. And in terms of test results review, we added a lot of features, speeding up the process for engineers. We created a whole bunch of integrations with this standard project management tool, set test case management tools. Then we improved our dashboards, building analytics on top of historical data that was collected by ZipRunner. And yeah, so basically ZipRunner became very popular among our dev teams and among our customers. So our customer customers, engineers and test automation engineers started using ZipRunner as well. And I believe that's when we realized that we want to try to move further as a standalone product company maybe. So we wanted to bring ziprunner out to the world. And these days, in fact, these days ZipRunner is more than just a reporting tool. So we have to tie a lot of aspects of testing together in one place. And we have our own open source test framework, for example, we have our own mobile device farm, we have our own managed scalable selenium grid. And this is basically, you know, the things that most of the organizations are using. So they have to, well do some sort of web testing so they can buy services from companies like BrowserSec, LambdaTest and sometimes they use real devices. Manual keys are usually using real devices but nowadays it's all automated as well. So you need something to tie everything together and basically this is where zebranu comes into the game. So basically the reporting and analytics part of ZipRunner is something that ties all of those things, all of those aspects together in one place under one umbrella.

I love that story because it's not a product that was just made up to say, oh maybe there's a need for this, let's create it and put it out in the field and see what the result is. It was based on what you actually were seeing from multiple clients and then you baked it into a tool for in house and then realized it could help other people. So I love that's how it originated.

Yeah, so we're actually talking. So we still, it's not like, you know, we thought that we have enough data and we are fine with what we have and we are going to evolve Zebra Runner in a certain way. So we are doing a lot of, you know, we are doing a lot of interviews, we are talking to developers, to engineers, to product managers, product owners, et cetera, all kinds of people from the industry to understand what kind of issues they are facing in their automation process, in their, well, software delivery process. And yeah, so we are trying to hear their needs, trying to hear about their pains and well, that's what we do, you know, to get an idea of how we should evolve Sib Runner in the future.

Nice. I know a lot of people listen to the show, a lot of guilders are really into helping make products better. So how do you do that? Is that a formalized process? If someone says, oh, I have an idea that I think would, I would love to have in a reporting tool. Could someone reach out to you? Is there like a website or a URL they can go to to submit feedback?

Yeah, absolutely. So basically it would be worth mentioning that ZipRunner is offered. There are three options how we offer ZipRunner and that's important in scope of question you asked. So basically ZipRunner is offered as a forever free open source community edition and it's basically Open source on GitHub and we have some repositories on GitLab. Everything is public there. So you can Go ahead. You can submit an issue or what you can do there. You can find a channel that you can join. So we have our own Telegram channel, we have our own Slack workspace. So basically we do instant messaging with our community members as well. So that's one way of reaching out to us. And then there is a cloud offering of Zebrunner which is based on zebrunner.com and there is a form. Basically you can fill out this form if you have like a broader question about Toast Test automation in general or about particular need. So you can reach out to us, describe your situation and we'll get back to you, we'll schedule a call and we'll be happy, you know, not to talk just about the zbrunner, but about how you do automation in general and see if we as a company providing a solution can help you to deal with this, with this problem, this issue.

Nice. Now I know there's other solutions out there as well. You have things like Allure and Report Portal. How is Zebra Runner different? Like if someone's trying to decide, oh there's all these solutions, which one should I go with? Anything that you think Zip Runner would differentiate itself from these other, not make it better or worse. Just wondering like if you, if someone had a certain need, Zebra Runner may match that need better than these other solutions.

Yeah, that's a very good question. So ZipRunner is more of a live dashboard rather than a static dashboard comparing to for example Allure. So we keep a lot of our analytics is actually built on top of historical data. Everything that we store on ZipRunner. And so we used it to build analytics, to build predictions and trends. So this is one thing that differentiates us from tools like Allure. And we rely heavily on machine learning which is not something that you see often. And we do a lot of failure classification with machine learning. So basically a lot of things that improve engineering productivity, the speed up process of test results review are built on top of machine learning algorithms.

Is that functionality built into the open source one? Is it only for the paid solution?

Unfortunately not. So this machine learning is something that's only available for our cloud customers. So we are thinking about open sourcing the machine learning part to open sourcing to open source. So that is probably something that is going to happen. We are still considering it but well instead of machine learning we have a lot of just non probability based algorithms that are also used, for example to track known issues. Well yeah, to do auto linking based on historical data, to link test executions, to test case Management system, that kind of stuff. So the core functionality is still in the open source. There are not so many features in cloud option that are not available to open source and those are more, well, I would call those enterprise grade features because aside from for example machine learning, we have a thing called project. So basically the concept is very similar to Jira and this is really helpful when you operate in as part of the large organization running, well, running multiple projects at the same time. So all of those projects can coexist in the same workspace. So it's basically like a workspace inside a workspace. So this is not something that you have in an open source and usually this is not something that is relevant for smaller organization running one product, developing one product. So some of those features and as I said, those are more enterprise grade features and all of the core functionality is still available in the open source offering as well.

Nice. Now you mentioned that it works with Jira. I believe it has a bunch of different integrations. Yes. If someone doesn't find an integration, how hard is it for someone to implement their own integration with ZipRunner if they had some other type of solution that's currently not available?

Well, it's probably important to understand how ZipRunner works to answer because there are two types of integrations when it comes to ZipRunner. So basically the very first type of integration is something that actually allows you to push data to Zbrunner. And our initial goal was to make this process as simple as possible. And that's why we created a bunch of out of the box integrations. We call those reporting agents. And right now we have integrations for the most popular Java frameworks. So those are the ones working out of the box. In GS World we have a support of Cypress. We're going to announce this reporting agents very soon. So basically just wrapping up the documentation. If you're doing automation in. Net, we have any unit support. Python engineers can use our Pytest engine. And basically if you something that I didn't mention or something that is not listed on our website, all of this reporting stuff is built on top of the REST API. So there is no magic behind it. We have a REST API guide allowing you to create your own integration. So if you're using some test framework that is not currently supported by ZipRunner it is really easy to create your own. So we provide a guide and in fact if the integration is demanding, we also offer our own help so it can be a shared effort creating this sort of integration. So that would be the one integration the first kind that actually allows you to push data to ZipRunner and then as a reporting analytics tool. ZipRunner itself is integrated with a lot of other tools, well from different categories. Those can be project management tools like Jira, Test case management tools. So we have an integration of, for example with Testrail, we have integration with QTest, we have integration with popular Jira plugins such as X Ray Zephyr. Integration is around the corner. We are also integrated with the most, well popular with the major vendors in terms of Managed Selenium, Grid. So we have integration with BrowserStack, Source Lab, LambdaTest and more actually. And if you're willing to become our partner, we have partnership agreements with many of those providers. So basically usually how it happens is that, well, representatives of their teams reach out to us and we basically just discuss what we want to do on our end and what can be done on their end. So both their customers and our customers and users can benefit from such an integration. And basically that happens like a collaboration of two development teams. But anyways, we have a lot of public APIs, so if you want to, well like if you want on your own to incorporate your own tool, for example, if you have your own in house bug tracking tool or something like that, we'll be happy to provide an API guide. So this is not something that's available publicly because every request is special when it comes to this kind of integration. So you need to reach out to our team, our support channels and basically this can be done via the channels I mentioned before. And we'll be happy to help, we'll be happy to provide API documentation. And once again, if this is something demanding, if you're not the first person to ask us about this integration, we'll be happy to provide our own help and our own developers to help you create this integration.

Nice. So basically we've talked all about reporting, but when I was researching Zebra Runner I kept running across the phrase handles parallel test running. So Zebra Runner is a reporting tool. How does it help with parallel testing then?

Well, there is one more aspect of zebrunners. We have a thing called Launcher and basically Launcher is a user friendly UI that allows you to actually schedule and execute test runs. And so there are two things that are probably important about launchers is, well, the one thing that we have, what we have right now, it's relative, relatively simple, it only supports the limited number of test frameworks because what Launcher does, in fact it actually creates a level of obstruction between a user and the particular test framework you are using. And it Allows you to configure your test runs by UI and that's when you can control the level of parallelization. And that will be one thing and another thing worth mentioning about Launchers. And well, this is something really exciting. This is something we are, well, hopefully will release by the end of this year. So that would be like a whole new, well, a brand new revision of launchers. And what we want to do, we want to build. Well, we want to still keep this layer of abstraction between a user and a test written by developer and it's going to be built on top of containers. So basically we want to run tests in containers and well, this idea of containerized environment makes things simpler for us and for engineers. So basically if you have, let's say a Docker image that needs to be used to execute your tests and you can basically, you know, like do some sort of configuration on ZipRunner end and then you can schedule your test runs by this UI with this special ui. I would say that it's not something that is really easy to explain in a few words, unfortunately, but the main idea that we want to, you know, we want to make sure that test automation engineer doesn't need, you know, to go to Jenkins, for example, to control parameters of the test launch, etc. So basically everything happens within the same ecosystem, so you don't need to switch between systems a lot.

Nice. So I first heard about Zebranum when I was doing my new show. I came across you acquired Korina and you mentioned Korina very briefly earlier in this episode. So could you just explain really quickly what is Karina for folks that maybe said, oh wow, I've never heard of that.

So Carina is basically a test framework that is built on top of testng and it basically combines a framework for API testing. Well, web testing. So yeah, we basically just added some popular features to testng to have like the framework that would cover them all. Yeah, so as I said, API testing. So for example, some teams used Rest assured for API testing. So we kind of incorporated some concepts of those to Karina. And our idea was to create test framework that can solve well, if not all, if not well, address not all of those concerns, but at least most of those concerns related to different kinds of testing. So this is the idea that was initially thought when we started to work on Carina. So this is something that we still use daily and we have decent community around it. So this is not something that only we use. A lot of people around the world are using Karina and they're contributing To Carina. And from our experience, it turned out that it's really easy, you know, to get started with Karina, especially if you are not an experienced test automation engineer, because you have everything in one place. You don't need to learn two or three different tools to test APIs and then to do web testing. So basically you have everything in one place under one umbrella.

Nice. And it's also open source as well, correct?

Correct, yes, it's 100% open source. There is no paid version of Karina, nothing like that. It's completely free.

So the kid, I know you work with a lot of customers that actually use zebrona. Any use cases or anything interesting you see in your clients using Zebron for that really are working for them or any success stories you'd like to share?

Yeah, absolutely. So in fact I can share two stories demonstrating two different perspectives on ZipRunner. When we started our conversation, we were talking about how messy automation can be. And that's not always the case. Especially when you're starting a new project. You have a chance to do everything the right way, to do things from scratch. I have one story about such a scenario. So basically one of our customers, they just started to build automation and they started to use Zbrunner since day one. And in our experience they mostly use it as a collaboration tool. So this is basically something that is used by their, well, engineering staff, test automation engineers, manual QAs and developers. So basically they use ZipRunner as a shared space where they can, well, both review the results and then they can bring the developers in. They can see the artifacts related to the particular test execution to get an idea of what went wrong and speed up the process of detecting what is causing the issue for them. So it's basically it's an engineer team that is using zebrunner on a daily basis. So that would be one scenario. And another scenario is, well, it's more complicated, I would say. So there is one large company that is using zebrunner and it's basically started to using zebrunner when they had a lot of automated test cases in place already. So they had, if I'm not mistaken, they had like 250k test cases automated. And the thing is that, well, yeah, by this number you can imagine how large this organization is. And they were doing a lot of outsourcing for test automation and at some point of time they realized that they, well, they don't have an idea of what's going on with automation, how reliable it is and how different teams, how can they combine and aggregate the results provided by different teams in one place? And that's when they started to incorporate ZipRunner basically to keep everything in one place. And management is actively using those dashboards and some of those dashboards are pretty advanced these days. So for example, right now they can give you an idea of, you know, how. Well, not only how pass rate changes over time, but for example, in a way it can help you to understand what is the ROI of your test automation. Basically you can see how much new test cases are automated, how much of the existing test cases are unstable, so you can get an idea where you need to invest your engineering time and fix those problems. And basically after some time they were running those 250k tests daily and they were producing more or less reliable results and they're still doing that. So basically their ZipRunner for them is a tool that helps them to clear up this mess and yeah, make test automation great again, I guess. And yeah, they're happy using ZipRunner these days. So I would say that those would be two most representative stories about customer success with Zubrunner.

Very cool. Okay, Nikita, before we go, is there one piece of actionable advice you can give to someone to help them with their automation, testing, reporting efforts and what's the best way to find or contact you or learn more about Zebron?

Talking about finding me and learning about Zebra Runner, you can always go to zebra runner.com and you can find all of the links related to all of the products we support and maintain, such as Carina and mCloud, which is basically our managed device farm. Or you can join our community in Telegram or in Slack, or you can start a conversation on GitHub. You can get in touch with us and we'll be happy to chat. And usually our response time is really quick, so it's not like you'll have to wait for an answer from us for like days. So it's usually a matter of hours to get in touch with us. And once again, we're very supportive, especially when it comes to people who are willing to use Zebruyner if something is not working for them, like if they're not understanding something, we are there to explain things to them, we are there to help and usually we can give you an advice even on how to build and structure your automation. So one of the things we did, and that's actually going back to the first part of the question about an advice on test automation. One of the things we did, in fact, we were doing some sort of audit of test automation. So for example, people were coming to us and they were like, hey, we don't know if we're doing the automation the right way. Can you guys help us? So we're doing some sort of audit and basically we were given a recommendation like best practices in terms of creating tests so they can be easily paralleled. For example.

We were given advices on how to not end up with flaky tests. And yeah, I would say that you guys should really use reporting if you have a lot of tests. So this is something I said already. So if you have a lot of tests and if you're struggling with analysis process, if you feel like you are dealing with the same routine again and again and again, Repartner Tool is probably something that can really, really help you to make your automation effective.

Again, thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a358 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Labs awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.