Playwright and CucumberJS with Sonja Quartz Leaf

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be speaking with Sanja all about playwright, cucumber and all other types of automation awesomeness. If you don't know, Saini is the Director of Engineering at cloudburst SBC and owner of SQL Media llc. She puts the queue in quality with the background and structured query language, complex system architecture, site reliability, browser automation, video game testing, and media publishing. And she's an awesome musician as well. Since 2006, she has contributed to software releasing ranging from commercial solutions and reward systems to the desktop, native web and console software. And Sonia started helping entrepreneurs begin their own tech companies when she joined Cloudburst in 2020. And Sonya works from her home in Minnesota, so I'm really excited to have her. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step depth, from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit testguild.comSourceLabs and click on the exclusive sponsor section to try it for free today. Check it out.

Hey Sonia, welcome to the Guild.

Hey Joe. Great to be here.

Awesome. Great to have you. So it's been a while. You actually spoke at Our Automation Guild 2021 event on playwright and Cucumber, so I thought we'd dive in a little bit around that. But before we get into it, is anything your bio that I missed that you want the Guild to know more about?

No, I think you. I think you hit it on the head.

All right, so this is already starting off script here and that is, I noticed, you know, when you started you were, I think, a quality engineer. And it looks like you made a huge leap and now the Director of engineering. So I just need to know, what's that like? How did that happen? What's the role change? If you could dive in a little, that'd be awesome.

Sure. I've been making software for a long time. I joined a young company as a senior quality professional. I wanted to be their quality engineer and they liked how I led that practice. I'd had experience doing leadership elsewhere in terms of tech. As our company grew, we decided we needed somebody to help wrangle the engineers and they chose me. It's a great position to Be in. It's been fun to influence quality at this level rather than after the code's been written.

Absolutely. So how are you doing that? Do you have any tips for people? Because a lot of times when you step into culture like that, it's like, totally different when you're now the director that has to do it rather than someone that's actually implementing it.

Well, it helps that I'm already familiar with our products. I'm not just coming in at the leadership level. I know how our code works. I've been an engineer, I've made software, I've tested software, and I like to be in the testing land. But really, Playwright was actually one of the things that helped me help our team become successful. We've had pretty wide adoption, and folks are making better, more reliable, more consistent software out of that. I showed it to a couple of my teammates, and they went, oh, that's cool. And there's some other things we'll get into with it that. That are really shiny about some of the latest features.

That's actually what's going to be. My next question is, when you spoke at the event, you were using Cucumber JSN Playwright. So I want to know is you still using it? And it sounds like you are using it, and it's actually been successful.

We focus just on Playwright and Playwright test. We do that because we do Typescript everywhere else. And Playwright has started to package their own test Runner, which has a lot of the same functionalities Cucumber had. It doesn't have the easy extrapolation of user stories to put in front of my customer. But I can still get all that information out in my test run. And it was much easier to get my test or my engineers to adopt browser automation when I said, hey, just write your test. You can call out your steps right here. What's actually happening in one file, rather than jumbling the handful of files that are required to make Cucumber run.

Oh, cool. So it seems like it's more streamlined. Do you still get the readability that you would get out of using bdd?

You do not get the readability, but we found that most of those stories actually end up live in our issue tracking system. They don't end up in our code base.

We tried having it in our code base for a while, and it was good, except it was hard to keep up with. We're a small shop, but we move fast. It was easy to include it all in new feature sets, just having it as part of the unit test sets.

Nice so when you see developers, these aren't testers, the developers actually writing UI automation and other types of automation.

Yep. We've got a small test team of two and then I've got a handful of developers and there we're writing our own test code and doing our share manual testing too. We're still doing rapid feature development and there's stuff that even though developers write their own test, testers think differently. I think you and I know this pretty well and I've got a good crew that make sure that the developers are actually building solid code.

Nice. So you sounds like you're a smaller team and you probably release a lot then. I assume so. You mentioned manual testing. How much manual testing is involved and how often do you release software?

We've got any number of projects going on, any at a given time, but there's probably every Sprint. When we're on two week Sprint, there's two to three releases that go out with a team of probably eight developers, two testers plus myself.

So I'm gonna dive into the decision, maybe the people that missed the session you did at Automation Guild of why you chose this tooling. But I'm still on this track now of when you started. When we talked, it sound like you just started with the Playwright. And now, you know, it's been almost a year. Curious to know how much did your test suite grow by and do you have any pruning in place? Like did you make any decisions that you wish you did differently when you started than if you were to start now?

You know, Playwright has grown so much over the last year, we've just grown with it. One of the things they've added is visual comparisons. I used to rely on external third party tools or like there's a JavaScript tool I've used in the past called Backstop JS. Good open source product, but now that's all built in. Hillary is one of my engineers. They're the reason I actually know so much about the latest and greatest of Playwright because they've gotten super excited about the release notes that have come out and as soon as they offered parallel tests, that information was in front of my face, thanks to Hillary. So shout out to Hillary now.

They seem to be going like gangbusters over there with Playwright. Actually I did an event for applitools called Future Testing and one of the leads engineers was on that call in his session. They came out with some really slick functionality. And so let's go back in time when you started, it wasn't this slick and it's like you still were behind Playwright. So why, why initially did you choose Playwright?

Initially, I chose Playwright because I needed to write tests with multiple user interactions. I looked at Cypress, I looked at Protractor to play nice in our frameworks, but Playwright allowed me to create a browser context and then I could create another browser context. And they had their own unique caches and cookies. So I could have maybe I have an administrator user and an editorial user, and I needed those to interact. And rather than tear down a browser, store the cookies, restart the browser, stash the, create a new user, I could do them simultaneously. So my test writing got faster and running got faster. I didn't also, I don't know. You've written your share of Selenium, it's hard to wait for elements, and Playwright has that figured out. There's this fantastic method called waitfor Selector, wait for selector, everything goes through and it is fantastic. Figuring out how to get a dynamic app to click at the right time used to be one of the biggest challenges I experienced, and I haven't had that with Playwright.

Wow. So it's been almost a year since we talked. Still no issues. Because you mentioned that as well at the event. That was one of the killer features.

Yep, still no issues. Actually, I won't say that like, let's say I have an intake form that has multiple questions that are yes or no. Sometimes if I want to click yes, it's hard to find which yes I meant. Even though there's only one that's visible. There might be many radio elements that are in the domain, but they've got this new concept that's come out called the Locator API, which normally when you write a selector and you select an element and you identify it as a variable, it's got that copy, that specific one. It located that moment in time. But Locator, the Locator API, you write it just like a selector, but every time you say you do click, it goes and looks for the newest one. So that Locator API helped solve problems. Some of those item consistency issues that I hadn't seen many of it was mostly just when I had buried form fields, but the Locator API has solved that for me.

Nice. So how's the speed when it runs the test? Is it slower than Selenium in your experience? Just as fast? Does it all depend, I guess, how you develop it, the script?

A lot of it is on the tester. As far as browser, I think the speed for me is getting up and running. That was always the biggest hurdle, and hands down, Playwright wins. And they've actually made it really easy to get into pipelines. You don't even need to use their special GitHub actions that I was talking about at the Automation Guild. There's just an NPX install command to install the dependencies, and you can use it on any CI runner.

So let's not overlook that. Can you explain a little bit about how easy it is to install? Because even with Selenium now with myself, I'm like, where's the driver? I'm on a Mac, I don't know where to put the driver. It's all craziness. Is Playwright like that?

Yeah, Playwright is so easy. So I can type NPM install G Playwright. And now Playwright is available in my whole system. And I can call the Playwright CLI by just calling Playwright, and then I can type the word codegen, and that'll open up a browser that I can point to any arbitrary URL with a little recorder already going. And that will allow me to generate new test code built for this Playwright test runner that I'm so excited about. It's an interactive tool like the Selenium Recorder, but better. It's not perfect, but it's better than the Selenium Recorder. I remember right?

I know they have a newer version, but the Playwright cli, I think it had just come out when you did your session. When I saw a presentation like a few weeks ago, I was like, wow, I didn't even know this existed. Has it gotten better since when it first came out? Are they improving that as well, or is it the same?

They continue to improve it in ways like making it easy to set up your CI, making it easy to launch a specific browser. A lot of those features have been there since the beginning. It's been a lot of the operational, make it easy to do continuous integration, make it easy to write new tests. They have improved the Explorer. So when you bring up the recorder, it used to just dump out the code in a window, but now it's actually easier to. There's more to interact with. You can actually explore the domain. You can hover over elements and it'll actually tell you potential selectors that you can type that would work for Those elements just continue to improve. And their Slack channel is very active as well.

I know Selenium just came out with Selenium 4, and they have a lot of great features. Selenium is awesome. We're not dogging Selenium, but it seems like Playwright maybe does a lot more things under the covers, maybe because of the way it was designed and because of that, you could do a lot of other things that you may not be able to do with a traditional UI functional animation script. Is that correct assumption that I just made there?

That's how I look at it. It sure seemed to solve a lot of problems in one set of dependencies. And the more that it. The more complete that package has become. I'm no longer relying on an external test runner. I'm just using the Playwright test runner. And that gives me things. It gives me really good concepts like page object that I don't think was in there when we talked last. And that visual comparison tool too. That's been a big win for us.

I know of applitools, so just curious to know what is. Is this different? Is this the same? What's the visual comparison tool like?

It's along the lines of applitools. You have a baseline set of screens that are objects. It could be a specific element, could be a whole page. Then during your test, you take a screenshot and you compare. Use a compare function. You will have a tolerance of how much you allow there to be differences. And then you get a visual report at the end that shows you what the differences were. So you can go in and review. Similar to what I've seen with applitools and other products out there. But it's really nice to just have it in. I have one developer, that developer, Hillary, they were on a project, sort of on an island, and they needed to make sure that this calendar they were building, it was going to generate a PDF and they had to make sure that the PDF looked consistent every time. And we were actually able to use Playwright, generate and consume that PDF and display it, take a snapshot and use that as a baseline and then compare the next time they made changes, go through, test it again, feel good about it. It's caught things for us. It's been a huge win.

Wow. So I was going to say, how reliable has it been? Was it really? Usually pixel to pixel differences in those type of tools have been a nightmare. But it sounds like it's worked for you.

Yep. You have to allow for some tolerance. You have to make sure that the pages that you're testing on the animations are consistent. And maybe you have to. Maybe here you want to have a timer to wait until an animation completes, or something of that nature. If you have an image that's maybe dynamic or rotates, sometimes the same thing you can get with applitools. You Might get inconsistent overlays like over a Hero image or something of that nature. But we found that it is consistent as long as we tell it to look at consistent situations.

That's a good rule of thumb. So another interesting thing is we keep talking about how they're adding features and functionality to Playwright. We covered cli, which is somewhat new because came out maybe a year ago and they keep improving it. This visual piece, is there anything else? I think I noticed something about an API that's able to stop and start web services for you. They have a lot of funky functionality like that. Is there anything else that you've seen recently that you're like, wow, this is really helpful.

So that locator API was really exciting to me. The test runner allows for parallelization by default, easy serialization, and you can even have all your tests in one file very easily go in one order. Even though we know that's an anti pattern sometimes that's the way it's got to be. The page object model has been great and they have expanded their support for more languages. So you've got good Java support, C Sharp support, Python support, and that Python support really interests me for the future because that's where all the machine learning tools are. And with the support of the of a page object model and Playwright and machine learning, I think that's where my team's headed next.

Oh, nice. Very cool. So, but it sounds like you started off using Cucumber JS because your team uses Typescript. So is that your philosophy changing now?

Well, so I learned to teach myself Selenium in Ruby because that was the most accessible language back then. And then I worked at a shop that specialized mostly in PHP. When I joined Cloudburst, we were mostly in TypeScript and JavaScript was my gateway to TypeScript. I was able to wrap my brain around JavaScript pretty quickly and the patterns there. And then one day the virtues of typescript and strongly typed languages really clicked. And even though it's a lot of setup, sometimes it the benefits for the quality and consistency of not just our test code, but our application code. They're countless having that compared to if we were writing native JavaScript.

So how about Python? If you started using Python, how would that work for you?

Well, one of my first programming classes was in Python and I would go there only because that's where the machine learning tools are. I'm not afraid of Python. I've made my own Google bots before that speak to me, so it's not a scary concept. But I did the main motivator for me with JavaScript was getting to better understand the domain my team was working in.

Well, you use php, so I can see how you're not afraid of Python because PHP is a nightmare to program for. Sure. Any tips for going? A lot of people think, oh, I only know Ruby. They minimize something when they don't realize, you know, it's just a syntax. You know the fundamentals, it goes from one language to the other. Do you find that?

Yeah, once you know a language well enough, you can adopt those concepts to any other language. There normally aren't unique. These days in modern languages that most people are working with, there aren't too many unique constructs to that language. You can adapt what you know about Ruby, but you know how to what a do loop does, how that might work as a a closure or anonymous function call in JavaScript or PHP. It can be a moment to shift how your brain thinks. But once you learn that it's just, oh, different words that mean the same thing or different symbols that mean the same thing, it's pretty easy to adapt those skills. I even had a project where I was using the dreaded Pearl for the longest period and I ended up loving it because it taught me so much about where these program languages came from that I don't have to deal with some of the crazy symbols and syntaxes that are required to have successful Perl code.

Wow. Yeah, absolutely. That brings me back Perl for sure. So interesting. Just go back once again to the automation session you gave in February. It was Cucumber JS using BDD Behavior Driven Development. So it sounds like. Did you abandon BDD Because a lot of people always complain about bdd. It's like a. It's just extra overhead if you're not using it for communication. What is one of the reasons? It sounds like you not abandoned it, but it seems maybe like it's not forefront like it may have been.

It was one of the pieces of our process that was hard to bolt on to all the other things we were doing with coming out with three releases every sprint or so. Having BDD on top was a challenge because we weren't using it for communication. My project managers loved it, but they didn't pass it on to the client and they understood the project good enough. So then I was only doing it. It was a helpful exercise for me to understand the products as I was entering into the company. But once I was in this leadership position and I had to get mostly engineers to adopt automated browser testing, it aligned very well, with some of the latest updates to elevate the behaviors. In my browser test with the Playwright test runner, being able to call out to the steps API in the test reporter gives us that same sort of information. We know it failed when I clicked book appointment.

How do you know when to evolve? Sometimes people maybe get stuck in like, oh, this is the toolset we use, we're going to stick with it. It sounds like you're a little more not open minded, but more you're able to adjust quicker. Maybe I'm just used to working with huge enterprises that just get stuck in the mud, but any thoughts on that?

You know, it is a different experience working at a nimble company. We do have a number of senior staff that have made software for a while. So when we do make changes, we do it carefully. Sometimes you just know a tool is right for the time it solves the problem I need solved right now and it solves it in an elegant way. We wanted to make sure it wasn't doing too much just because I liked the idea of the toy or of the project or the tool. The tool actually solves a problem. So if I see a problem being solved and it makes it easier for my team to build better, more reliable code, that's an easy thing for me to sell to Leadership.

You seem like someone that if you actually use a different tool, a technology, and it worked better, you'd go to it. So in that type of mindset then you know, because there's no right tool for everyone. It's like you said, it depends on what works for your technology, what you're trying to do. Who do you think Playwright is a good solution for? Is there any specific functionality a certain technology you think it's just like this is like perfect for this type of situation or scenario?

I think it's really perfect for applications that A need quick integration tests. B if you have multiple user interactions, that was the place where it really excelled for me first and continues to excel for me where I have a producer and a consumer type user that need to interact with each other. And that has been where Playwright continues to shine. And it was great for me as just a automation test engineer. Really easy for me to adapt and I really liked not writing all. If you've ever been frustrated by writing weights in Selenium, I'd say give Playwright a try.

Okay, Sonia, before we go, is there one piece of actual advice you can give to someone to help them with their Playwright or automation testing efforts in general? And what's the best way to find or contact you.

The best way to find or contact me is through LinkedIn or Twitter or Instagram Sonia Q. Leaf or Sonya QQL the as far as advice, don't worry about testing everything. Get some tests done that is like get some tests done and make sure that you have some level of quality at least to get going. To learn the API like doing Creative Endeavor, you have to build a number of these browser tests to really feel confident and really feel comfortable with it. So write a few, see where it starts to work for you, then go back and flesh out those areas as you learn the APIs more thanks again.

For your automation awesomeness. The links to Evident value we covered in this episode. Head on over to test guild.com a372 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.