261: WebDriverIO and Extended Debugging with Christian Bromann

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with Test Automation, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of what used to be called Test Talks. Still Test Talks, but we'll soon be called Test Guild Automation Podcast. In this episode, listen up and get an inside look in the life of Christian Broman, the maintainer of Webdriver IO. Christian shares some pros and cons of using Webdriver IO and some extended debugging features he's been working on for the Dev Tools team at Sauce Labs, plus a whole bunch more. Check it out. Test Talks is sponsored by the fantastic folks at Sauce Labs, the cloud based automated testing platform that eliminates the need to maintain your own Selenium grid and test infrastructure. Try it for free today. Visit testalks.com and click on the Sign up now link under the Homepage Sponsor section.

Hey Christian, welcome to Test Talks.

Hey Joe, nice to be here.

It's great to finally have you on the show. I've heard a lot about what you've been doing Webdriver IO. I've been watching a lot of videos on you with doing presentations in WebDriver IE, Appium and automation in general. So those are all the types of things I'd like to talk about today. But before we get into it, can you just tell us a little bit more about yourself?

Yeah. So I'm Christian. I'm working as a software engineer at Sauce Labs. I'm working here in the DevTools team and that team is trying to build capabilities on top of webdriver, which allows me to work on exciting things like extended debugging or performance testing. As you mentioned, I'm also the AC rep for SaaS at the W3C, so I'm representing the company in the standards committee. I'm also, as you mentioned, the maintainer of the Webdro project and I like to be involved in the overall foundation that is behind that project. And yeah, as you mentioned, I like to speak on conferences whenever I have something interesting to tell. So that's me.

Very cool. So, Dev Tools team you mentioned you work on things like extended debugging. Is that what's built into Sauce Labs? Do you develop that? What's your involvement with that?

The whole topic came up when I finished my college and back then I built a driver for smart TV that was based on the Chrome DevTools protocol. I got really exposed to that protocol and learned a lot about things that it provides and and I try to apply that technology or that protocol to the offerings that we have here at Sauce. So with extended debugging, we created a different communication channel to the Chrome browser in order to allow us to get not only performance data, but also network activity and console adjacent like console log data for the customer so that he can have a better insight and what is happening in the browser.

Very cool. I think one of those new insights, I think your team recently rolled out how to do specifically with performance testing, is that correct?

Exactly. We recently released a new performance offering that allows customers to test the front end performance of the application. This is difficult from usual performance tests as you heard here in the industry that are more we related to load testing and there are already a lot of great tools that you can do to load test your APIs and whatnot. But this is specifically designed for front end performance testing, meaning it gives you an idea about how quick the application is being loaded, how quick it is rendered. It also gives you the performance score based on the Lighthouse calculations. So we use Lighthouse in all the great dev tools and Google technologies under the hood and to give you the data that you would usually not able to get with just basic webdriver.

So what's that data? Are there certain metrics that you're able to pull out that WebDriver can't?

So as we have a different like an additional.

Parallel connection to the Chrome browser, we are able to control the communication between the Chrome driver and the Chrome itself, which allows us to start tracing the browser and get all.

Log information about whatever is happening in the browser engine from Chrome. And we use the tools that are provided by Google like Lighthouse in order to make sense of all this data. Because those tracing lock data can have a lot of payload. Like there could be piles of like 6, 7, 10 megabytes that you would usually not be able to use. So we use those tools to then get metrics like the performance score from Lighthouse the time to Interactive first meaningful paint those kind of metrics.

Cool. So you mentioned a lot of times there's tools out there that are made for performance testing. Why then should a tester care about performance testing rather than just say, oh that's a separate team that handles that. I don't need to worry about it.

So it's all adds to the conversation of shift testing to the left. I mean you could have a dedicated team that is only focused on performance and some big companies have that. But if you want to provide value for your quality assurance team in order to not only provide a functional test security, but also performance test security. You can just use the additional capability that we provide in order to have a second layer of testing which is in this case a non functional testing factor, which is performance.

Awesome. Sounds like you do a lot of things at Sauce right now, but you also are the main maintainer of webdriver IO. Like where do you get the time? How did you get involved with webdriver IO? What's the backstory with that?

So the backstory I think started when I was in college, like when I started college. My when I started college, I think when my university teacher had this project for us and we had to test it. This was one of the requirements back then and it was a single page application and I had a hard time to figure out how I write unit test for the single page application. So I googled around and found almost nothing on the Internet and then stumbled upon the project that was called webdriver js, which then was renamed webdriver at some point point later in time. But that project basically was one of the first Node JS implementations to run browser automation, to run browser automation in Chrome and Firefox. I think back then and I used it heavily and after I was successful using it, I continued contributing to the project because it kind of feel really interesting to contribute to open source and get involved into Node JS development. And that's how I started learning about Node, learning about open source. And it made me a lot of fun. So I did that next to my studies, next to my other work that I do. And till today I have a lot of fun working on this project.

Very cool. So I'm sure a lot of people always ask or curious to know when would they use one tool or implementation over another of a certain framework. So are there certain situations where webdriver IO is really made for like it has a sweet spot.

Interesting question. I think every tool, as you mentioned, provides some advantages and comes also with some disadvantages. The things that I'm most proud of in the Web2vel community is the people behind the project. Because there's some. For instance this Gitter support channel where we have, I think by now 4,500 people that help each other out constantly. Like we have people that heavily involved on daily basis and it's really specific to one single project. So whenever you have some question there, there are definitely people that can help you out. And otherwise I think webdevo next to Protector is one of the tools that give you a full fledged framework to your hands. So it doesn't require you to implement anything that has been implemented before. It comes with all the things that you need to build a scalable and sustainable test suite. And there are a lot of people now that are constantly interested in evolving the project and it's just fun to work with that.

Cool. So you mentioned it has a lot of built in things for you so you don't have to reinvent the wheel. What features do you think most end to end testing solutions should include that WebDriver has?

So apart from many other opinions, I think reporters.

Have to be really specified for end to end tests because if you run if you use a unit test framework like Mocker or Jest and want to run end to end tests with it, the reporters usually don't tell you which capability has been running that test. So it's really difficult to then figure out hey, the test is failing, but what browser, which browser has failed my test. So I think reporters have to be really specific to end to end test. And then another plugin or plugin type is the service, which is also kind of unique because it allows you to enter or to implement special functionality in the test lifecycle. Let's say you want to execute something before all test run. You can do this with the.

OnPrepAreHook. If you want to do something specific before every command or before every test, or after every test after every command, you can do that as well. You can do not only run specific code, you can have that code run in asynchronous fashion. So the test actually waits until.

That execution has happened. So it really gives you the opportunity, for instance to put data into the database that is unique for your test case in order to test something. Or there's services out there that highlight elements whenever you interact with it. So there are a lot of opportunities.

That you can use with those kind of folks.

Cool. So Christian, one of my pet peeves is maintainers main contributors to these open source tools. Obviously I think they're the cream of the crop of automation engineers, so they know the things that automation engineers are struggling with. So I'm always curious to know why people blame the tools when their tests are failing. I think in one of your videos, Appium for Couch Potatoes, you mentioned that a lot of folks blame the tools when in fact it's something else that's causing the issues. In your experience, what are these other things that could cause issues rather than just blame the tool or the framework?

Yeah, in the end to end test world, I think we are testing A wide variety of components. It's not only the front end, the back end and the database, it's much more. It's also the driver, the tool that we automate our browser with, and the browser itself. There are so many areas where a factor can happen that we don't. People that use those tools don't usually know where the error comes from.

Since you're using automation tool like WebDav IO to automate a browser and something fails, they only usually blame the automation tool because they are the opinion that, well, my application was working when I was developing that feature. So in the wide range of.

All the error areas that can happen.

The decision to blame the error on the framework that you use to automate the browser is usually always the first one because it's the one that you don't, that you don't know or that you have no influence on. And it's the one that usually promises you that whenever you use Set tool and you say you want to click on the element, the click should work, but they totally forgot that the click has to go through the whole browser engine and the click has to also be worked through the browser driver that is automating the browser in the first place. So there are a lot of unknowns that the user doesn't usually is not exposed to.

Any tips on how to get to these unknowns of what actually caused underlying issue? Is this built in like does the dev tools, is this what they work on all the time to try to correlate it? It failed and here's why it failed because of the network was slow or something like that.

I mean they're like all those areas you can. It's hard to figure out where those errors come from if you haven't been, if you haven't seen them before or if you don't have an idea where they could come from. The dev tools definitely can help us to make some sanity checks on specific things to identify the error area that WebDriver itself as a protocol could not provide us. So yeah, one of the things that is happening in the WebDriver protocol area is that we now want to connect ourselves or allow to connect. Will allow the front end, the automation framework to connect to the DevTools interface behind the browser, which will help all these frameworks like Selenium or Webdrive IO to get more information out of the browser to make sure to give better ideas about where the error comes from.

Awesome. So how evolved are you nowadays with webdriver? I o I just saw an announcement fairly recently that A newer version came out. I'm always seeing updates from WebDriver. I o how many contributors are there to WebDriver IO?

So right now we have.

Four or five people in the technical steering committee and a bunch like 20 people in the. We call it project maintainers. If you read over our governance model you will see that we currently order or we bring or we order our people or contributors to two categories. One are the project maintainers and one. The other one are technical steering committee. Becoming a project maintainer is pretty easy. You just have to show like some decent sort of interest into the project and then you be give you get like access to the repository can can help us triage issues and provide pull requests. And then the technical steering committee are people usually members that have been have shown a lot of involvement and are then.

In the. You know, can then decide which direction we want to go. And we recently had our first technical steering committee meeting where we just. Where we looked into all the areas that we want to explore in the project and we found a couple of that are really interesting which are for instance dev tools service that we want to improve which is similar as we had as I mentioned before. It provides a second connection to the browser and allows us to do a lot of things on top of WebDriver. And that tool can really help us to improve the automation and allow for instance things like stabbing and mocking requests of the browser. That is usually a highly asked feature of people that do browse automation. And then we have other things like we want to provide a fiddle platform that where people can put in their code and allow to share reproducible example for the community or for their colleagues. And in general.

Provide the easier onboarding to the project by generating sample files that get you up and started pretty quickly.

Awesome. And I'm looking at the project roadmap and it's pretty ambitious just framework support. So how do you tackle this? Is it by committee, by vote or is it just someone has free time and they have an area they're interested in so they'll start working on it?

Yeah, we talked about that in the committee and we ended up saying well we have all these initiatives that we want to tackle and whenever someone has time he is welcome to help out on this. It's at the end of the day an open source project and none of us is really full time focused or earns money with that project. So we can only ask if anyone who wants to have time, who wants to learn Node JS wants to be involved with the project. Project can help us out. So we don't have a real ordering. Everyone who is interested in any specific topic can explore it and help us moving it forward.

Cool. So Christian, you also mentioned that just recently that you were working on making it easier for people to contribute and I think you even created a. You talked on this topic. What specifically is being done to make it easier for people to contribute? And do you think there's something now that's prohibiting people or making it harder for people to want to contribute?

I think one of the biggest achievements to make this easier was the move to a monorepo. So we moved all the web developer projects and all the packages that are contributing to your test run cycle into one repository, which makes it really easy if you have changes to multiple sub packages to see how this has modified the test runner or has modified the.

Webdriver itself. So having everything in one repo is helped a lot to simplify that process onboarding process and then we created a general governance structure that showed people how being involved into the project looks like and how I can be more involved into the project. And as you just mentioned, we built up a roadmap that shows where we want to go with it and where they can be involved. So this is also a big plus that helped us to get more people on board with this. And then one of my goals that I have for the future is, and I tried this before last year I think is to have onboarding sessions with people that wants to get involved. So I would pair with whoever wants to pair with me on the project to fix a random bug or work on some new features in the project. And I would put myself apart one hour a week and get that person trained up and work on him together on a specific feature.

Very cool. How popular is Webdriver IO? Is it becoming more popular? Can you tell by the download numbers? Is it growing exponentially?

Yeah, actually really know that because this is one of my things I take a look on every day almost because it's just for me, if I see that the project is growing, then it excites me more and I'm more excited to work on this. So it is still like, I think Selenium is still more popular than revdeo by far. And there is Cypress now that has got more and more traction into the automation space and that is exciting. So I think besides Selenium and Cypress Repair is at least in the JavaScript area, one of a more popular framework. Yes.

So there seems to be a lot of NPM packages that can help you with automation. Are there any other packages you think.

Engineers should know about in terms of automation or in terms of browser automation? I think there are a lot of other projects that allow you at least in the Node JS world that allow you to automate the browser but I'm not aware of any like there's of course the selenium bindings that you can use to automate to build your own test suit and there sub projects like I think Knifewatch JS are there. I heard not good things about Protect unfortunately. So it seems that that project is not being maintained and specific for Node. I heard a lot of small project but none of which that have like a high development velocity.

Yeah I heard same thing with Protractor. I heard it's good for small projects but people weren't wowed by it.

It apparently is still used internally at Google but there's no one really from Google who's continued working on it.

So I guess that leads to maintain a burnout is what I'm thinking is how do you avoid maintain a burnout? Because it seems like you've been doing this for a while. You're obviously a busy guy, you do a lot of sessions, you travel around the world, you work for Source. So like how. How do you avoid maintain a burnout?

Good question. I think one of the things is that I need to. I often set myself apart from the project so there's times where don't even look at it for a while like for a day or two in order to get a little bit, you know, space and and then I'm always trying to find new exciting things that I can build with that framework and with that tool to help other people or to help people in new ways to improve their lives in the QA world or in the deaf world. So I'd always try to find new challenges around the project that helped me to continue stay exciting as part of in the support channel. I often, you know, find myself don't giving the appropriate answers when I have to and that is something that I have to work on and I have to you know this I think will train me in my career to be always, you know.

Self, self sustained and always.

Looking at myself and how I give answers. So those are things where I also try to improve myself.

So I'm also curious to know like a lot of times people don't contribute to open source projects and so I think if more did less people would get burned out and wouldn't be just on a few committers. I think this would Open up a lot of doors for people that are contributors. Maybe I'm wrong, but is this by you contributing to webdriver IO, is that what came brought you to the attention of Source Labs? Or is it just because you worked on Webdriver IO, you started having relationships with people and then started recommending you for jobs like at Sauce.

So yeah, this project literally opens all the doors that I currently have entered in my career. I mean starting with working on the project. I approached Adam Christian, who was back then engineering manager at sas and he gave me the internship to work for SAS during the summer. So that got me the job at Sauce in the first place. While being at SaaS, I was able to join the W3C and was able to contribute to with my ideas by having the knowledge that I got from working on the project. And last but not least, since I contributed the project to the OpenJS Foundation, I'm now able to help building up a foundation that is the umbrella on so many great open source projects. And with that you meet so many great people that have the same passion as you do. So I always recommend everyone who's, you know, who is currently in college or already graduated or has already his career and never worked on open source that he, you know, that is one of the things that can really help you to do whatever you love in the software engineering area or in tech in general.

Yeah, very cool. I was at sourcecon this year and it seemed like a lot of new hires of people that have been working in the community for a while. So it's like almost in your best interest to contribute to Open Source. Even though they may not be monetary that value it actually I think is in the back end if people realize just put the kind of contacts you make.

Yeah, absolutely. Like, I had a lot of questions if I would open a fundraiser for the Rep project. I never was like this project should never give me any monetary incentive. It always should give me incentive to or interest in, communicate with people and help people out in order to find new challenges around browser automation and test automation.

Berkund, as I mentioned, we were able to speak a little bit at saucecon this year. There were a bunch of different announcements. Is there anything that you were excited about? I know the performance piece was one of them. Was there anything something to do with Headless? Did you work on the Headless technology that they came out recently with?

I've not personally worked on it. I think Headless is still a really exciting project because it lowers the cost of running tests in the cloud and scaling up your test suit and browser automation framework, but I think there are currently still challenges around how we leverage that platform in order to be really efficient. I've been working with the project manager on example, where I was spinning up like 100 browsers, headless browsers at once and having you know what having watching my project so whenever I do a change it would run all those headless containers without having to recreate the session. So that allowed me to really run end to end tests at the speed of unit test almost. So there are a lot of interesting challenges around there and I think next to headless and performance there will be new exciting things that come out from SaaS labs and I'm really looking forward within the next quarters.

So Christian, does the DevTools team actually use WebDriver IO to test what they're developing? Like do you actually use the tools themselves when you're working at Sauce? Like do you use webdriver IO to help you with your day to day job on the DevTools team?

Yes, we use WebDrio in order to test our drivers internally. So we have, we don't use the Chrome driver that you can download from the project page. We have our own Chrome driver, our own wrapper around Chrome driver that is managing the connection to the browser. And like any of the drivers we have to test this driver really carefully and we heavily use wrapped here to make sure that whenever we release this driver that our customers have like.

That.

The tests of our customers don't start to fail.

Okay Christian, before we go, is there one piece of actionable advice you can give someone to help improve their WebDriver IO automation testing efforts and let us know the best way to find or contact you.

So the best way to find and contact me is over the GitHub support channel that we have for the Web Developer project. You find this on the website and the best way to improve your skills in web developer or browser automation in general is to start contributing and being involved into the projects. Because it also not only teaches you how the project works, how the framework works itself, it also shows you how test automation works in general. And that is something I always advocating for. That is really important if you want to find the errors that are hidden in all these areas, if you want to spot them, and if you want to find out where they come from in order to quickly fix that.

Thank you Christian for your automation awesomeness. The links to everything of value we covered in this episode head on over to testtalks.com,261 and while you're there, make sure to click on that. Sign up for a free trial link under the exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. So that's it for this episode of Test Talks. I'm Joe and my mission is to help you succeed with creating automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com for full show notes, amazing blog articles, and other automation awesomeness.