Root Cause Troubleshooting with Bnaya Peretz

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with BNE all about Root Cause Open Source troubleshooting with Puppeteer and Playwright. Test Vinay is a developer at Testim and an open source believer. Has a real lot of experience developing, especially for open source. And I think it shows, especially with this new release of testim's Root Cause open source tool to help you troubleshoot your Puppeteer and Playwright tests. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly. Flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey B', Nai. Welcome to the Guild.

Thanks Joe. I'm happy to be here.

Great to have you. Before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

Well, I'm a full stack developer for around 10 years now. As you said, I'm enthusiastic about open source and user experience. Developer experience. Also, I don't only like to code, I also like to hike and do, you know, scuba dive and do other stuff and. Yeah, very cool.

So it seems like you've been very busy at testim. I know Testim seems to be releasing new features now every month. I think the latest one is around Root Cause. So before we get into it, I believe Root Cause the new feature has to do with Puppeteer and Playwright. For the folks that don't know, maybe talk about a little bit. What is Puppeteer and what is Playwright?

Sure. So Puppeteer and Playwright are browser automation libraries that they're very good browser automation libraries to help you control browsers programmatically. Now Puppeteer started as a Chromium only library, but they added also Firefox support, it's from Google. And then Microsoft did Playwright, which is for all of the modern browsers. It's for Chromium, Firefox and Safari Webkit. And these libraries are in use not only for end to end tests, but also for browser automations in general. But I don't know if mostly for end to end tests, but also for end to end tests. And they're very powerful. It's easy to write tests with them, but sometimes it can be difficult to debug the test that you write with them. If the test fails, great.

So you are a hardcore developer. Are these tools you actually use yourself?

Yes, of course. I use the Puppeteer for browser automations and test for, I don't know, in Puppeteer since the release, I think it's like, I don't know, three years or something like that. I think it's actually the thing that sparks this browser automation. Modern browser automation is the Chrome headless mode. I think it's only out for four years. It lets you easily run Chrome in Dockers and also in low memory servers like Lambda. And then it enables you to write end to end tests easier also on Dockers and in small machines.

So I think Playwright, is it a fork of Puppeteer and if so, are there any benefits of using Playwright? You've seen yourself using either one of those, so.

Well, basically Microsoft stole some of the original developers of Puppeteer from the team in Google that worked on Puppeteer. I'm not sure exactly, probably started as a fork, but they did a lot of work. Probably now two different animals. The API is very, very similar, but the API is also similar to what you would see in Selenium in Node Selenium, because, okay, you have a browser and then you have a page and then you can click on stuff. So the API is similar, but behind the scene to support additional browsers. They've added a lot more work, not only on the JavaScript side, but also on the native side in the browser to make Playwright. But again, since then, also Puppeteer added Firefox support and they also think to add more browser support.

Nice. So I guess anytime I see a vendor investing in open source tools, I assume they're becoming more and more popular. And I think Testimon also created a recorder for both Puppeteer and Playwright that makes it easy to create tests for them. So have you seen an increase in folks using Puppeteer and Playwright and you see that as a trend, like you said, more modern type of automation tools for modern needs.

So first I think there is increase in general in writing end to end tests. It's people becoming more aware of the importance of writing tests and people are going out to look how to write their tests and there are like many tools and yes, Puppeteer and Playwright are a lot of people are choosing to use them and sometimes you use them as your main tool, sometimes they're like a small block in your entire test infrastructure. For example, you would also need A test runner. You don't only use Puppeteer because Puppeteer is a node library. You need the test runner to run it inside. So you need to pick the test runner and then you need to pick the automation framework and you can see more and more people are looking for end to end test solution. Again, great.

So you did mention that sometimes it's hard to debug these types of tools. So let's get into now the Root cause release, I guess that you have, what is the root cause? What's the official name? The Root Cause Open Source kit, What is it called?

Yeah, so it's called Testing Root Cause. Okay, Testing Root Cause, that's the full name and basically it's a node JS library to help you understand what's going on in the puppeteer or playwright test that you wrote. And best way to understand what it does is to first to think when you don't have root cause, like how your workflow going to look like. So you have a web app and let's say you have 100 end to end tests and you have a test runner. Let's say you pick Jest. Jest is a very popular node JS test runner. And then you made some changes to the web app. You invoke Jest and it runs your 100 tests. And when you deal with this amount of tests, you probably usually run them in parallel. And you run the browser in headless mode because you don't want a lot of browser windows to pop on your screen. It can also be on the CI. So you even can see what's going on now because the change that you've done to your web app, let's say five of the tests now failing and one of the tests fails because it can find the selector, one of them, it fails because the click failed. Each test has its own reason for failure. And now you need to understand what you've changed in your program in your app that failed the test. So what I would do if I don't have root cause, I will run locally the failing test one after one. I will run it in non headless mode and then I will try to understand, I will see the test playing and I will try to understand what's going on. Maybe I'll add a breakpoint, maybe I will add explicitly, I will add breakpoint screenshots and then I will look in the screenshot I will save it to the disk and then I will look what happened in the specific moment when the test. So what we are doing with the root cause, we Automatically will in the case of jest, you edit the integration using jest level. And then we will automatically test the screenshot for every step in your test. Every step mapped usually to a puppeteer command. And it will log all of the console log that we see in the browser. We save it in our database. The console logs screenshot for each step, network logs and other and the stack trace of each step. And then after the test run ends, you can view all of that information on your test. You get list of the steps and then you can click on the steps. And if a step fail it will be marked in red and you can click on it and you can see a screenshot. You can see also if you use a selector in your test like you were looking for an element or something like that, you will see highlighted elements. And that's actually save you the need to sometimes will even send you the need to run the failing test again. You'll run the test, you'll see the five failing tests usually will open root cause through the cli. So you write root cause, show you type the test ID and then it will open the viewer and you will see the test information. And again that can save a lot of time to run Test 1 after failing Test manually. And it saves a lot of frustration because sometimes the test will fail. I mean in the CI it will fail, but locally it will fail once in five times. And when you record that info all of the times, you can see the state of the page and screenshot on the run that it will fail in and then you can fix it to not to be flaky again.

Very cool. Yeah, sounds like something that's really, really beneficial for folks to use. Is there any overhead of using it? If you're saving a lot of screenshots, is it being saved locally or how does that work?

So the screenshots are saved locally and there is some overhead because we are adding additional calls to puppeteer like for each command that you do, we add screenshots from the check that we did. It's not very significant. And we are also monitoring that very important for us. And you can turn it on and off easily. I mean it's an open source project, you are all very welcome to try. And if you have any issue like you see that your tests are much slower. So please do open an issue on our GitHub repository. We want it to be fast as possible.

Absolutely. So it is open source, but is created by testim. So is there anything that becomes paid at some point or if someone consumes this library is it going to have to be paid to unlock certain features.

So the root cause local when you run locally, it's completely free and again it's open source. You can have your own copy of the code. And we do have cloud tier. It means that you can save the results in the cloud. If you run it in the CI, it can be very convenient. Also locally, if you run something and you want to share a link with someone, you add you need to register to the testim cloud and then you provide your token and then you can choose to upload the results. Cloud there is pretty big free tier of like 3 million artifacts. Each artifact roughly maps to a screenshot or something like that. So 3 million is a lot. But you can also. You don't have to use it. I mean it's very useful also without that feature.

Nice. So what test runners does it work with? Is everything that playwright and puppeteer work with. I know you mentioned Jest, but I know there's Mocha Chai other type of runners as well.

Yes. So now support ourselves, we have package for Jest, we have package for Mocha which is the two most popular runners. We have also integration API. It means if you have your own runner, you can import some functions from the package and then you can write your own integration. And if we will see demand for any specific asset runner, we would add support over to that. Probably we would want to add to a runner called ava. It's also I think it's the third next most popular popular test runner in Node. But it also really depends on the community, what the community will ask from us.

Nice. So how does it to get started with it, how do you instrument your test? Is it just calling the library? And then I think I saw something like a date feature. A date function you had to create.

Yeah. So the example you are talking about, it's the Jest integration. So first it's very simple in the case again I will bring Jest as example and then I will talk about Mocha. So in Jest you need to add us as after and ingest config file. You can add code that will run before each test run and that's where you launch your. If you use a library called Jest Puppeteer which is not related to root cause. I mean we work with it, but it's not directly related to us. You would add it and then it will launch the browser for you just out of the box. Will not provide you with will not found Puppeteer browser for you. Puppeteer it will Add the browser for you and then if you add just puppeteer and you have the global page object, you will add us in the as in the documentation, you will add us in the set of files after N and then you will add us also as a jest reporter. And then that's the only steps you need to do to integrate. And the thing you mentioned, you need to create unique run ID and you need to pass it to the reporter and to the end. So that's why you need the date, the current date. It's currently built that way with Jest because there is some limitation in the API to pass these values. But I hope it will manage to overcome that to make the integration even simpler. Now the integration is like, I don't know, four lines in your Jest config. We will hope to make it, you know, even one or.

Cool. So you did talk about how much easier it is to debug. I want people to feel the pain though. Without the solution, how do people debug their tests now? Is it just adding like log files, log statements between each call to a method or something?

Yeah, something like that. Actually, going back over my example I gave before, let's say you have, I don't know, six call inside your test. You need to manually add the screenshots between them and to save it into, you know, temp step one, step two, you need to give them name by yourself and then you need to go to that directory and look picture by picture. You don't have any like nice viewer for that. And also if you run two tests and you use the same, you run two tests in parallel and you have two tests writing to the same directory, you will have overwrite. So it's like very not comfortable. There is many ways to debug. I mean there is like many ways to debug code in general. You can also use debugger, you can debug jest test, you can debug mocha test. But with root cause, you don't even need to do that, at least to get the main problem in the test.

So I think most testers are familiar with getting screenshots for each test or between steps when it fails. Is any other power of the visualization that you're doing in the root cause test and root cause library that helps visualize issues or maybe common causes over time or anything like that.

So if you use in root code cloud part of the testing cloud viewer, it's something also that is available on the general testing product, not only in root cause, it can filter or group test by common failure. It also have report weekly reports of your switch runs and this kind of information.

Awesome. So it sounds like with the visualization part you can easily triage failed tests and say this is a scripting issue or this is a real bug. I guess it would be helpful that way.

Yeah, it really is. And we might also add these features if the community know if we have a lot of features you think to add and we need the community feedback to see what is more important and what we should add first. And we can add some aggregated information on test failures also locally, not on the cloud for your current run but we need to the feedback from the community to prioritize what to do next.

So yeah. Do you have any features you'd like to see added but you're just waiting for the community to get back to.

I wish I had the feature that will fix the test for me or something.

I'll write my own app. No, that's AI self. Okay. Actually that specific thing that testim automates have this kind of feature like suggestions and AI based analytics on test. But I think the next thing I would want to, you know, I have too many things that I. I want also to do on the product that I can't even pick something specific right now. What we want to do is really to reach out to as much as developers we can and to. And to see that we really help them to solve their problems. And then to see maybe we, I don't know, maybe something in the visualization can become, you know, can be better, maybe stability issues. That's our focus.

So you know, I know you were joking but I guess because this is in the testim ecosystem, if someone actually has a license for testim, is there a possibility at some point where it can say okay, let's utilize an AI algorithm now to on the fly be able to self correct an issue that's being flagged in the. The root cause.

Theoretically it can happen. It's not something that we are working on right now, but it definitely can happen. And we have many interesting ideas. I mean we have also here, you know, this specific project Root cause is an open source because we believe there is many things, you know, we want to give back to the community and we can give value in that way and we need to find the balance what can still be open source and what would be, you know, the value that testing brings. And also we want to keep that product independent as you say. We were not want to put in any vendor lock in and this kind of thing, we want to give the good product standalone. People can use that without being locked in any way. And. And then you can see how we can add features, maybe premium features, you know, that won't be open source.

Cool. So besides this, I think you also, I read somewhere does something with the HAR file. I know a lot of people with HAR files, they do weird things like they can get performance metrics out of it. So where does the HAR file come involved in the root cause open source library?

Yeah. So one of the things that we collect information that we collect about your test is record HAR file of all of the network activities during the test run. And then we have an internal viewer that you can view inside the root cause viewer. You can see the HAR file and also you can download, you can view it per step and also you can download the whole HAR file and then load it into your. It's a Chrome devtool. The network tab is also HAR file viewer. You can just drag the whole HAR file there and then you can see the HAR file and then you have all of the performance metrics inside. I mean you can see how much request took and all of that. We still don't have any specific feature to give you to analyze the HAR file and to give you insights. Okay, the thing or slow or something like that. But it's definitely something that we can do. And then, you know, it's also, it's a question we, if you want, if we right now want to focus on, you know, end to end tests and then, you know, performance testing. It's in some areas it's overlapping in some areas it's, you know, it's a different domain.

And I guess along the similar lines, you mentioned some about console logs. Is console logs just like a printout of what happened along with the screenshots or is there anything more involved with the console log functionality?

The console logs are not the node JS logs, but we collect the console logs from inside the browser. As if I would open my dev tools and then I will click on things or there will be any unhandled exception. I will see that in the browser console log.

Oh cool.

And yeah, when the browser is headless browser or in the server, I don't have access to that. So we collect them and we save them as part of the test. It's also, you can see them. So you can assume that if you, let's say there was a click, you choose the click step and then you would see an error in the console log. So you can assume that the error is related to the click to the click step.

So when you're looking at a screenshot, does it show you then that this is related to this? Like if there actually was an error within your console logs, would it flag that visually for you as well right now?

No, because it's hard to very like to correlate exactly if a click calls or what elements related to the to the console log. It's definitely something, you know, that we can if we see that a demand for that or we will have exact, you know, model of how to make it, how to connect all of the points. Yes, it will be make sense to add that. And something that it's worth mentioning that if you click on a button so you won't get only the screenshot, you will get also a highlighted inside the screenshot, you will also highlight the button that you clicked on. So this is something that you do get.

Nice. So are there any other features, part of the root cause that I may have missed that you think people need to know about?

So I think it's worth mentioning that you can still run your tests in parallel. I mean you're not limited by root cause in any way. I mean you can run your jest test as usual. If you run in parallel if you want, it will still work.

So are there any limitations on that? Brings up another point that's a good one. It doesn't affect parallel, but there's anything that people need to be aware of, like hey, if I use this then I can't do this feature or I may need to be aware of that this is going to impact something else.

So first again we use it ourselves. We have some tests for that, but we want to find out if there are some use cases that maybe are not working with that very well. We're still not sure and maybe there can be some problems if you try to do multiple page tests or this kind of test. So maybe in some cases it might be problematic. But again if someone will bring up, you know, issues with that and then we can see how we can make it work.

So I don't know if we even addressed this. How long has this been out? How new is it? How recent has it been released?

I guess so I think we made, you know, that we make the repository public, I think two weeks ago, I think something like that, I don't remember exactly. And then we have the first version out. We are working on it, I don't know for the past three months or so and we are still working on it and so yeah, it's relatively new but we have a lot of experience with tests and we bring our experience from the testing automate product to try to make this product the best we can basically.

Cool. So I just want to take it up a high level. Since you are an experienced developer and you personally contribute to other open source projects, I think sometimes testers may be not afraid but maybe don't know how to stop contributing to open source. This might be a project that they'd be a perfect fit for. So how hard would it or any recommendations you give to someone on how to start contributing to open source projects like the testim root cause.

So I think the biggest contribution to open source it's not necessarily code or pull request. It can start with participation and feedback. Participation in to open an issue. Let's say you had a problem and let's say you managed to overcome it or you're not sure how to instead of like you know, throwing away the tool and not use it to open an issue and try to provide the good reproduction information and collaborate with the maintainer of the project to solve the issue and that also maybe other people have issues and you maybe you had that issue and you overcome that issue. So maybe try to help other users in the GitHub repository, for example in the issue and that I think it's like the entry level but it's not only entry level you do that you know all of your open source involvement also include a lot of stuff now about code. It's you need to usually to contribute to open source. You don't only need to learn to know the language that say that's written in notes. You don't need to know. You know, you also need to have some domain specific knowledge to really understand what's going on. And it's something that you can acquire over time. I mean you start to work with a project, you see how it, see how it works and then you maybe you debug it a little bit and then you can understand how and where you can improve it. So if you are already experienced developer, yeah everything can happen quicker. But also something that can really be helpful, it's to contribute documentations or to improve documentation. It's every project have readme files and documentations inside the repository so you can even add new sections or if you've done, you know, good intro about that or you, I don't know, you've done something that you would want to share, you think it will help other users of the library too so you can open a post to edit and then again to collaborate with maintainers to see how would be the best way to bring it into other users knowledge.

Great advice Arkipane before we go, is there one piece of actual advice you can give to someone to help them with their automation developing testing experience? So from maybe a developer's point of view, automation best practices or something and what's the best way to find contact you or get ahold of the new testim root cause so you can find.

Us at GitHub at GitHub testimil root cause it's GitHub.com testimony testimio root cause.

Please come to visit us there and please if you have any question or any problems open an issue. We really need your feedback and your involvement and if you like our project also we will happy to star us or fork the project in GitHub and also feel free to reach out also by email. We will leave also all of the LinkedIn information on the description of the podcast. Something that I find that were very helpful for me is to split common steps in tests. You can say it's like I don't know, more prepared steps or common steps of test to separate functions and you can even call them not inside your test body this function. You can call this function in the before each or before so on of the test. It really helps to make the test organized and then if you change the setup phase of the test, you don't need now to update.

You need to update it in one place. It's the same with code. Basically when you write coded automation test, it's usually it's the same as code.

Thank you Biner for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com 323-and-if you haven't heard or if you haven't registered already, head on over to secureguild.com and register for an online conference dedicated 100% to helping you succeed with security testing. I think it's one of the number one skills you're going to need in 2020 and beyond. So if you really want to help your prospects of getting hired in a new position or you want to grow within your career or find a career that really is a growing field that I think is an excellent place for testers to be involved in that is security testing accelerate your learning efforts. Now it's an online conference dedicated 100% to security testing from a tester's perspective. Register now@secureguild.com so that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.