Fast and reliable cross-browser testing using Playwright with Arjun Attam

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be speaking with Arjun Adam all about fast and reliable cross browser testing using Playwright. Arjun is a program manager on the Playwright team at Microsoft. Playwright is a new browser based automation library for end to end testing. It also focuses on test automation that is fast, reliable and capable enough to automate the modern web. You don't want to miss this episode. Check it out. The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial, visit testguild.com sauce labs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Arjun, welcome to the show.

Hey Joe. Thanks so much. Super excited to be here.

Awesome to have you. There's a lot of buzz actually going on Playwright, so I thought you'd be the perfect person to talk to. So, at a high level, what is Playwright?

So Playwright is a new cross browser automation library for end to end testing. Right. And the goal with Playwright really is to provide a single API to developers and test so that they can automate their web applications across the three major browser engines today, Chromium, Firefox and WebKit. Playwright is sort of another entrant into the web automation space and it's trying to actually introduce automation that is fast to execute, reliable to execute, and highly capable to the needs of the modern web. That's really the basic idea behind Playwright.

Awesome. So I want to dive into that for sure. One quick question. You say this is new, but is it based on Puppeteer or is the same people that create Puppeteer there? So there's been a lot of confusion, I guess around Playwright is a puppeteer. Is it Puppeteer? You know, 2.0, is it based on Puppeteer Core? And now it's going to be diversion between the two. Two different tools.

Yeah, sure. So you know, playdead did start off as a fork of Puppeteer. You know, Puppeteer for those who are not aware, is an old library to automate the Chromium browsers with the JavaScript API. And they've also now introduced experimental support for Firefox. So Playwright did start off as a fork of Puppeteer and you know, that has been sort of the starting point when it comes to the API. That's been exposed to users, but in a lot of ways, you know, Playwright has a different philosophy. For one, paywright is explicitly focused towards end to end testing and that influences the API that's made available to developers and testers out there. For example, actions in Playwright would auto wait for the element to be ready. Now waiting for an element to be ready is a common pattern that developers have to explicitly write in their code. And Playwright does that automatically. This makes it easier for you to write tests, makes your tests more reliable, and this is generally more important in the space of end to end testing. Playwright is also adding new capabilities. For example, you know, Playwright has added support for WebKit and so you can test your web applications across chromium, Firefox and WebKit, which sort of covers the three major browser engines out there. And it's also adding newer capabilities. You know, for example, we've introduced a better API for file downloads. You know, we've introduced better handling for out of process iframes and site isolation. And you know, the goal really is to actually, you know, take Playwright closer to the needs of developers and testers out there for the purpose of end to end testing.

Nice. So would you say Puppeteer is more towards developers then doing headless type testing, where Playwright is more towards end to end testing for more tester related workflows?

Yes, I think from a Playwright's perspective, we spoke to a bunch of developers and testers out there who were using various cross browser automation tooling for automating their web applications in the context of end to end testing. And we found out that there were gaps in being able to run these tests across multiple browsers or being able to use a simpler API and simplify the learning curve for using automation. Right, and Playwright is sort of focused on those problems, you know, from the get go.

Awesome. So if someone is using Puppeteer, can they migrate to Playwright? Do you recommend they do it or does it matter? I guess Playwright, like you said, is one of the main differences. It does work with other browsers. So maybe if someone wants that functionality, they want to merge, are they able to migrate then from one to the other easily?

Yes, I think that should be possible. You know, we've seen some, you know, teams sort of migrate from Puppeteer to Playwright. You know, at the end of the day, it sort of depends on, you know, what your practice. For example, if WebKit is an important browser target for you and you want to test how your web applications would look on iOS Safari, you can emulate that experience with WebKit and Playwright, for example, that's an important target for you. Migrating to playdead makes a lot of sense. There are lots of API changes that we've introduced. For example, Playwright is centered around the notion of browser contexts. Browser contexts are isolated environments that are built on a single browser instance. And so, you know, it's possible for you to actually create concurrent tests that are completely isolated from each other and sort of execute in parallel through browser contexts. Browser contexts also enable you to actually emulate scenarios where, you know, you're changing the device viewport or using permissions or geolocations across multiple pages. And so concepts like this are sort of important to, you know, understand before you actually doing that migration so that you can actually utilize the best of the Blade API.

So you mentioned a bunch of functionality I want to dive into. I know you actually just released version 1.0, maybe, maybe it was like a few weeks ago. And what's new in version 1.0 that you've improved upon already on the initial release of Playwright, there have been a bunch of improvements.

You know, if you sort of look at the quick timeline, you know, late January is when we open sourced Playwright and sort of announced version 0.11, I think, and last week is when we announced 1.0. So that was about the 6th of May. Between January to May, the repository has been super popular, GitHub, and we've been extremely fortunate to actually get a bunch of user feedback on the repo. You know, we've received hundreds of issues from, you know, users, early adopters, who've sort of shared, you know, painstaking details into their use cases and what they want to achieve. With Playwright, you know, this has included issues around new functionality, for example, downloads, how to process iframes, how browser contexts work. This has also been about reliability. The web is a very complicated platform, and web applications tend to be very, extremely sophisticated these days in a beta manner allowed us to actually reach a wide variety of applications that sort of battle tested the reliability of the tool. And so there are a bunch of improvements that have happened there. And then finally, the overall developer experience of using Playwright has improved a lot. With the 1.0 release, we created a documentation website for you to actually understand the core concepts of Playwright, look at simple snippets that actually walk you through the examples, Go through the API reference if you want to. We also introduced better integrations for continuous integration environments. Your CI CD pipelines are an important target for you to execute Playwright. And so with the Playwright 1.0 release we created, we released a GitHub Action for Playwright. So if you're using GitHub Actions as your CI CD provider, you can get started fairly quickly with Playwright tests. The API and new functionality reliability improvements and the overall developer experience have been sort of the three verticals that have had major improvements since the initial release in January.

So everything I've read about Playwright is that it always says it's more. It helps you create tests that are less brittle. So can we go over maybe the main features that help support why it would be less brittle? And then I want to go maybe talk about maybe some competitors or some other alternatives and see how it compares. So when you say it helps write tests that are less brittle, can you give some more examples?

Yeah, for sure. Actually, maybe take a step back and just understand web applications today. Modern web applications need to be extremely rich and responsive. Developers out there are building single page applications that do a lot more in a browser tab than web applications used to, let's say, a decade ago. The rich and responsive behavior of web applications is essentially powered by handling asynchronous events. If I'm clicking a particular button on a page, stuff happens on the page. Your web applications are not isolated, they're powered by your backend services. You're talking over the network. There are lots of asynchronous events that are happening inside an application, and being able to predictably automate these things is hard. You might run into cases where a network call takes slightly longer and your UI shows you a loader and you know, you can't actually jump into automating it. Or, you know, you might be sort of expecting a new pop up window to be released and, you know, you might have to sort of switch context and automate that window. You know, there are tons of these things that are involved in building web applications today. You know, this asynchronous nature actually creates it difficult for, you know, any sort of automation to work and sort of reliably automate things that are in your application. Which is why developers and testers out there are actually struggling with automation. And, you know, the whole idea of like brittle or flakiness sort of comes into the picture. Right. I think frameworks and drivers out there are trying their best to do what they can, but given the requirements of web applications today, it's a really hard problem. What Playlist tries to do is that it tries to approach the problem from, you know, by introducing intelligence defaults that sort of make things easy out of the box, but then also reveal capabilities required for more advanced Scenarios. So, for example, clicking on a page is a very obvious action that lots of automation scripts out there are doing, right? And before you can actually click on a particular element, you want to be sure that the element is visible on the page. You want to be sure that the element has stopped animating. You want to be sure that the element is ready to receive pointer events and, you know, you can sort of act on it. Now, Playwright tries to do, you know, as I was saying, Player tries to introduce some intelligent defaults and tries to check the element is ready for you, right? So it goes through a bunch of checklists and ensures that the UI is ready for you to actually like, you know, execute your actions. Now this is important because, you know, when I'm sort of in the context of testing, I'm trying to define a user scenario that I want to test. I want to get to the user scenario, right? I want to get to the user scenario and say that the user clicks on this particular element. You know, my user scenario does not explicitly call out that the element is ready, the element stopped animating, you know, the loader has gone. Because as users we're just implicitly hardwired to, you know, wait for these things, right? And Playwright tries to do that. However, in a bunch of these cases, you know, it's very hard for Playwright to automatically figure out what's going on. And you know, for those Scenarios, playwright has APIs which are again, cross browser but give you more control. And these APIs come in various shape, but one popular way in which users can use them is the wait for APIs. So given a particular event or a signal, I can wait for it before execute my next steps. So I can wait for a particular selector or UI element to be there on the page and ready, or I can wait for a network request to have completed. We've taken the approach that you have these capabilities available to you in a way that you can express the state of your application and wait for it. And we've tried to provide you enough capabilities so that common issues that cause brittleness or flakiness can be, you know, avoided. That's sort of the idea behind, you know, Playwright giving you auto weights or, you know, just giving you a lot of granular control over execution.

Nice. So there is a community around all these different automation frameworks and tools. One of them obviously is Selenium. They have a new W3C standard. So I asked my audience, you know, some questions they'd like to ask. One of them is, is how does this compare against SELENIUM not to say one's better than the other, but like, is there a scenario where you think Playwright would be good for and maybe Selenium would be better for another type of user case?

Selenium is the most popular automation library out there and it's really established the automation space. I mean, the last decade or so has been about, you know, Selenium sort of becoming extremely popular and really creating the whole world of automated UI testing for the web. And so it's been widely successful, it's created a large community of users around itself. And you know, Playwright is trying to sort of take an approach which is slightly different because we feel that modern applications need another short automation. And that's to really say that if you're building web applications that are using sort of newer web platform APIs or newer techniques using JavaScript framework that are sort of helping you build single page applications, the difficulty in automation increases if you're using frameworks like Selenium. And we're trying to sort of eliminate some of those problems by the techniques that Playwright is using. Having said that, Playwright is, is barely four months old and we recognize that we are nowhere close to the maturity of Selenium and the broader ecosystem. So this is really an attempt at trying a few things out, sharing it with users, understanding whether things are working out for them, being able to validate the design decisions that we've taken with real world problems out there and just seeing how the underlying driver can evolve. It's really an attempt at pushing the web automation space in total and contributing to the overall story by experimenting and introducing some newer decisions and getting to a point where, you know, we can actually add greater value to the broader web automation industry.

Great. Now this is just from my experience, I'm not saying everyone's like this, but I used to work with developers that were supposedly JavaScript developers, but they could not do automation with Selenium. No matter how hard I tried, it just didn't fit into their workflow, their ecosystem. And like you said, there's a lot of things you need to do for waiting for elements for in order for it to be reliable and they'd get frustrated. So it almost sounds like Playwright would be a good solution in that situation. If you have developers that are JavaScript experts as well and that wanted do testing that this may be a good solution for them because it sounds like it'd be like a seamless type of technology that they can incorporate with their existing tools. Is that valid?

Yes, for sure. We have consciously focused on JavaScript for now, you know, because that is the language in which web applications are built today. We understand that, you know, automation is hard for everyone. For developers and testers, it's not easy to automate web automation. The whole world of web automation is just full of hard problems. And Playwright is an attempt at, at solving it at sort of a more foundational level where you have more APIs and more capabilities to sort of play with, with the intent of making your automation scripts less brittle and sort of more capable. And that's sort of the starting point that we've chosen to explore with Playdead and 1.0 release.

Great. Now, this is a question from Twitter that they wanted me to ask. Since Slim came out, Slim is older. Technology has changed. Front end technology is really JavaScript is like you said, we have React Angular js. It sounds like almost starting new, maybe why you started a new framework for this to handle these newer type of technologies. But they want to know why did you decide to make Playwright a proprietary solution rather than working with the W3C to improve on the WebDriver protocol?

That's an interesting question because, you know, we don't look at Playwright as a proprietary solution. You know, it is an open source project on GitHub. Everything that we do is completely open. Our issues are a place for us to get feedback and our source code can be analyzed by anyone who sort of is curious. We're using a very liberal open source license and, you know, we are very welcoming when it comes to, you know, contributions to the repository. So Playlit is entirely built in the open and there is no proprietary or closed source element, you know, behind it. We're also sort of moving into using the issues more to actually lay out the roadmap and externally communicate our planning and thought process. So we've been very super open in building Playwright so far. We will continue sort of doing that from a broader standardization perspective. Like, I think it's a matter of prioritization. You know, web automation is full of hard problems. There are tons of like, you know, problems that, that developers and testers are facing out there. And, you know, we've chosen to sort of fix some of those problems and chosen a particular approach in solving those problems, right, which is to release something and make it available to users out there and actively listen when it comes to, you know, their problems and their feedback in sort of using it. So I think from a maturity perspective, we want to sort of invest into the driver, make sure that it works well, and hopefully then can we actually get to A point where, you know, it can actually add value to, you know, the whole standardizations conversation.

Awesome. So another feature, I think tools like Cypress became popular. A lot of people use Cypress because A uses JavaScript, but also it's able to do things with the network like get network timings. It sounds like Playwright is able to do something similar. Is that true?

I'm not exactly aware of this particular capability, but you know, great control over your network is extremely important. You know, in this age where your web applications are making, you know, asynchronous XHR or fetch calls, with playdead, you can actually intercept network requests. And being able to actually automate your network and know understand how network activity is happening in the browser during your web application is a, is a pretty important area. And so Playwright actually gives you the ability to intercept network requests so that you can actually abort them. You know, if it's, let's say a request to an analytics website, which you don't want in the context of your testing, you know, you can actually abort that request. You can also modify requests. You know, we've seen cases where developers want to send in an additional header in their requests while they're testing stuff so that server can actually respond in a certain way. Or you can also completely mock those requests request, right? So instead of actually sending them to the server, you can create a mock response object and send that to your application. You know, all of this is possible through Playwright. You can also wait for specific requests to happen or specific responses to happen so that you can actually, you know, have granular control over the state of your application. Again, you know, at the risk of sort of reiterating, this is possible across the three browsers. So you can do this inside Chromium, you can do this inside Firefox, you can do this inside WebKit with the same API.

Now, another thing I've heard folks struggle with with modern applications is Shadow doms. Does this Playwright handle Shadow doms fairly easily?

Yes. So Shadow doms is actually one of the newer capabilities that we introduced. You know, in the last four months, based on user feedback, we've seen web components and Shadow doms being, you know, picking up really quickly. And you know, one of the early user feedback that we received was users wanted our selectors to actually automatically pierce Shadow DOM so that, you know, they could actually select things, you know, select elements that are actually built using the Shadow dom. You is something that we introduced in Playwright. So our selectors automatically pierce shadow routes. So, you know, if you're using let's say class names or other CSS selectors. If you're using ID attributes, they're fully compatible with the Shadow dom. There are also light variants of these. So for example, if you don't want to actually have, you know, this behavior out of the box, you can disable it. We've tried to actually provide intelligent defaults wherever possible so that it's easier to get started and accomplish your goals. And if your goals involve, you know, using the Shadow dom, Playdy tries to do that, you know, in a very manner. Manner.

Great. You did focus on working with continuous integration. Is there like an API or is it easy to run from CI systems?

Yeah, so we recognize that, you know, when it comes to automation, you're using it in local and cloud environments, right? In your local environment, you're using your automation driver to run tests or write new tests. In your cloud environments, you know, which can be in your CI CD pipelines, you're actually running your tests and trying to get a result, trying to get a red or a green build before, you know, you merge a bull request to create a new release. And so CI is an important area where having a CI integration actually helps teams rely upon end to end testing. Right. For my broader team to be convinced and feel confident about our automated testing, we use our CI CD pipeline and run tests in our CI CD pipeline to ensure that everything's working. To make it easy, we've released a bunch of recipes which are actually available across multiple CI providers. So we have documentation on getting started with GitHub Action. Travis CI, Azure Pipeline, CircleCI Appware. These are super popular and these are part of sort of the broader developer experience of using Playwright. We've also released a Docker image. So if you're not using one of these CI providers but can use a Docker container, you can get started with cloud deployment of Playwright with that Docker image. We've also released a GitHub action for playwright. So if you don't want to sort of dive into the configuration and have something that just works out of the box and you're using GitHub Actions and use our GitHub Actions added to your actions YML and you know, get started with running playwright on CI.

Very cool. I know another thing that JavaScript developers I tend to know, tend to like, they like Jest. Does this integrate with Jest and other common test runners?

Yeah, that's actually a great point. So, you know, Playwright is built with the idea that it's, it's extremely modular, it's focused on being an automation driver and work nicely with, you know, other parts of your testing stack. Right. A test runner is obviously a very important part. And we've seen users use Jest or Mocha or, you know, Jasmine or other sort of test runners out there. And we've added a documentation page on how you can actually get started with these things. Right. So if you're using Jest, for example, you know, you can actually use your before all or before each hooks to set up your browser, create browser contexts, and get your tests ready in a way that you can execute them in isolated parallel environments through Playwright. We've also seen a lot of open source activity test. Playwright is a very popular repository out there which actually tries to bring some of this configuration out of the box. So you can actually, you know, if you're using Jest, you can pick that preset, add it to your Jest configuration, and, you know, write Playwright test. Yeah, that's sort of an important part of the sort of broader CI and test runner experience, which Playwright fits into.

Very cool. And also another thing automation engineers tend to struggle with is parallelization. You talked a little bit about it, but what features makes parallelization execution easier? With Playwright, some feature that makes it easy to parallel your test easier. More easy, yes.

So the way this works is, you know, we've introduced this concept called browser context, which is really central to all your execution that happens on Playwright. The way to think about this is, you know, you basically have a browser instance of Chromium, WebKit or Firefox, which is something that you can launch or close with a Playwright API. Now, this single browser instance can actually be used across multiple sort of isolated executions through browser context contexts. Now, browser contexts are essentially a very cheap isolated environment that's created on top of an instance. So parallelization, yes, that's actually an interesting sort of problem area. So, you know, let's step back and basically look at, you know, how the various entities when you're executing an automation script are structured. Right. When you're automating a web application, you're using a browser instance. And this browser instance could be, you know, Chromium, Firefox, WebKit. You can launch these browser instances, you can close these browser instances instance. Now, Playwright introduces a concept called browser contexts, which is very central to the execution model. So one single browser instance can be used to create multiple concurrent isolated browser contexts. These browser contexts are extremely fast and cheap to create. So you can create hundreds of browser contexts and you can close them down each. Browser contexts can host multiple web Pages, right? So, for example, I'm using a single webkit instance, and I'm creating a browser context on top of that, and I'm running my scripts inside, inside that browser context. Now, because these browser contexts are isolated and concurrent, you can actually run parallel tests with a single instance of WebKit across multiple contexts. Now, context also introduce the ability of emulation scenarios. So every context can have a specific browser viewport, it can have a specific permissions configuration, it can have a default geolocation. So I can actually create a browser Context for the iPhone 11 viewport with a geolocation set to roam, for example, and give it a certain set of permissions and basically run my web application within that emulation environment. So you can imagine this case where let's say I want to, I'm creating a web application and I want to sort of run it across a matrix of mobile and desktop viewports and a matrix of geolocation or sort of permissions, and I can basically spin a browser context for each one of those parameters and run them in parallel. So a single browser instance, which is. Is generally very expensive to launch and close, can be reused in isolated environments by browser contexts. And you can basically use that to parallel execute your tests across a multiple set of emulation scenarios. That's built into Playwright. And everything that you execute in Playwright across chromium, Firefox or WebKit executes in a browser context. And it's pretty central to how Playwright is of build.

Very good. I know we're running out of time. I just thought of a user case that I had in my previous job. We had a radiology application where a radiologist would log in, look at a study. But then someone needed to test that that study was locked when a doctor was looking at it. So they need to bring up another browser log in as that user and make sure it was locked. So it was very flaky. Could context using Playwright help with a scenario like that?

Yes. So one of the things that you can actually also emulate in a browser context is authentication credentials. You know, lots of applications use browser cookies to, you know, define authentication state. And you can actually have two different browser contexts on the same browser instance. Would have two different authentication credentials out there. You know, all applications out there have a sense of, you know, permissions and user roles. So I could sort of log in as an administrator in one context and log in as a dummy user in the other context and essentially run my tests in parallel across these two different authentication credentials. Credentials. The key thing really, again, is to reuse the instance, right? So which means that you're not actually spending time launching a new browser or closing it and sort of recreating that again and again. You can do this expensive operation of launching before your tests execute and essentially create new contexts which are extremely fast and cheap to create and destroy in the context of your tests. So authentication credentials is an important part of the emulation scenario story, along with mobile viewports geolocation permission permissions.

Awesome. Okay Arjun, before we go, is there one piece of actual advice you can give to someone to help them with their Playwright Automation testing efforts? What's the best way to find or contact you?

Yeah, for sure. I think I would encourage everyone to find us on GitHub. So we are on GitHub.com Microsoft Playwright GitHub is central to all our product development and we'd be extremely happy to actually have new issues from you talking about your use case or what you want to accomplish with Playwright. If you want something more real time, we also have a Slack community where we're sort of talking with our users in real time and it's a way to sort of engage with the broader community. If someone wants to reach out to me personally, you can email me or find me on Twitter. My Twitter ID is Arjun Atum. That's a R J U N a double T A M and my email address is my full name with a dot in the middle.

So arjun.atamicrosoft.com thank you Arjun for your Playwright Automation awesomeness. For the 17 of value we covered in this episode, head on over to testguild.com.

And while you're there, make sure to click on the Try it for Free Today link under the Exclusive Sponsor section to learn all about Sauce Labs awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.