Fullstack acceptance testing using Serenity/JS with Jan Molak

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Yan Mala all about Full Stack acceptance testing with Serenity JS 2.0. Yan Jan is an international speaker, consultant, trainer and Full Stack developer specializing in continuous delivery and all those aspects of a software delivery process that helps you make your automation efforts happen. He's also the author of Serenity js Jenkins Build Monitor and other open source projects helping thousands of companies worldwide keep their builds green and deliver their process smoothly. I'm really excited to have Jan on the show. He's an expert in this area. You don't want to miss it. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Yann. Welcome to the Guild.

Hey Joe. Hey, thanks for having me.

Awesome. Great to have you on the show. Really enjoyed your session at last year's. Well actually was this year's Automation Guild. We get a lot of questions around Serenity js, so I'm really excited to have you on the show. Before we get into it though, Jan, is there anything I missed in the intro that you'd like to share with the Guild?

No, I think you've covered pretty much everything really. Basically what is I help companies all over the world to help improve the way they deliver software and and help improve the software as well. And as part of this work I tend to capture the things I learn in open source and then share those with community and Serenity JS is one of those projects.

Nice. So I am very familiar with Serenity, the Java implementation. I loved it when it first came out and I used it pretty much throughout one of my latest engagements and got really great success with it. So I guess at a high level, what is Serenity JS? Is it just a JavaScript implementation of the Java version?

So there's actually a funny story behind it. So years ago I used to work with John Ferguson Smart on Serenity BDD and we were client who had a number of Java teams and also some JavaScript teams. Now those JavaScript teams were slightly reluctant to use Java language in the Test Automation So they were wondering if it would be possible to implement the same patterns and same ideas in JavaScript. So actually that's how Serenity JS started those four years ago. So it starts as a little PoC to help a JavaScript team implement the same ideas on their project, but since then it has evolved to be a proper full stack acceptance statistic framework. Many ideas in it that have already been implemented in Serenity, but the way it does it is actually much more specific to the JavaScript environment. So you'll notice that the implementation of the screenplay pattern, for instance, is fully asynchronous. So you can use it to integrate various test tools that would normally require you to use promises or callbacks, and makes it in a way that you actually don't have to worry about those promises and callbacks too much in your tests.

Nice for the folks that have no idea what Serenity is at all. Why this framework? What's special about Serenity and Serenity JS that makes it? What are some of its strong points that someone may choose it for sure.

So the way I like to think about Serenity JS is that it's a next generation acceptance testing framework. Now, what do I mean by that? So you can think about Serenity JS as an integration framework so it doesn't try to replace the tools you already know and love like a protractor, webdriver, Axios, and so on. What it does, instead it integrates those tools to help you create fully full stack tests. So one of the key points here is that you can actually write a single test that would interact with your rest APIs, your UI, and even your domain models should do so. So that's one of the key selling points, that you actually don't have to write separate tests for those different endpoints. You can use the same language to express those different tests. Furthermore, you could reuse those building blocks you create to interact with those separate endpoints in single test scenarios to create hybrid tests. But integrating test tools is not actually the most important thing about this framework, I think. More importantly, it helps you integrate people involved in writing and reading those tests. Now, one of the challenges testers very often face is how to get other groups involved in writing and reviewing the tests they write. Now, I don't only talk about developers, but also business analysts, product user experience, designers, architects, and so on. Now, as you know, typical webdriver tests are difficult to understand to anyone but the testers themselves. Now, Serenity JS implements what we call the screenplay pattern. So the screenplay pattern is a way to express your tests using business language. So Screenplay pattern helps you to capture business vocabulary in your tests and then use the same vocabulary to use test reports and leaving documentation. Now, this makes it much easier for people with no testing background to understand what is going on in the test and what outcomes trying to verify. Furthermore, by framing your tests in business contexts, you make it much easier for people like release managers and product owners to understand the impact a given test failure have on their product. So I like to think about it as an integration framework that integrates both your tools and the people who use them.

Nice. Yeah. When I was trying to explain Serenity to my team, I used to say it was like a wrapper on top of WebDriver and Rest API, Rest Assured. And it really took care of a lot of the boilerplate code you usually have to create from scratch using Java, and it just made it a lot easier for people to implement and write automation.

Absolutely.

I assume it's the same thing with Serenity. So. So what about the BDD pieces? We'll get into the screenplay pattern. But is the screenplay pattern the same as bdd or is BDD you still use something to drive BDD within the JavaScript ecosystem and then use the screenplay pattern to create the code that actually implements the BDD steps.

Right. So BDD is more about the collaboration rather than automation. But what people typically use in the BDD context is they use tools like Cucumber, for example. So Serenity JS works with cucumberJS very well. So what you could do is you could capture your examples, your executable specifications in the form of Gherkin scenarios, and then invoke Serenity JS from within your Cucumber steps. So this gives you a very nice way to progress from the business vocabulary you've already captured in a Gherkin scenario to the business vocabulary you're capturing in your Serenity JS steps. But it's not just that. I mean, if you're a team BDD and likes Cucumber, then that's a very good way for you to use Serenity JS with Cucumber. But you could also use it with other test runners. So right now, Test supports Jasmine, for example. So what you can do is you can use Jasmine scenario to then invoke the Serenity Actor flow or Serenity steps. So Serenity is actually not limited to any specific test runner, and you can quite easily extend it to support other test runners that you might like.

So one of the great things about Serenity Java is that it could integrate with all these other frameworks. I believe Serenity JS also has the same concept using module, is that correct?

Precisely. So Serenity JS is a very modular framework. So While the version 1.0 was a bit of a monolithic framework that evolved from this proof of concept that I told you about in the beginning of our conversation, Serenity JS 2.0 is much more modular. So the idea here is that you have the core module that provides you with the screenplay pattern APIs, then you also have a number of other modules that integrate SerenityJS with your test tools and test runners and external reporting systems. So, for example, you'd have a module for Protractor that integrates Serenity JS with Protractor WebDriver, you have a module for testing Web Rest APIs that helps you generate Serenity BDD reports from Serenity JS and so on. And you could quite easily create those modules yourself to integrate Serenity JS with any other system that you might want to interact with, or any other reporting system that you might want to send your reports to.

Nice. Yeah, I know another popular framework. I'm not saying I want you to say which one's better than the other, but another popular one is webdriver IO. When someone's making a decision on what framework to choose is there, do you know how this stacks up against webdriver IO and when you would use one over the other, or what preferences would be?

So that's a very good question, Joe, and one that I get quite often, actually. Often enough that I actually did a little proof of concept to see if we could integrate Serenity JS with webdriver IO. So, like I said earlier, Serenity JS doesn't tell you what tools to use. I mean, it can integrate with pretty much anything. So right now it ships with support for Protractor. But the next big thing I want to do is I want to integrate Serenity JS with WebDrive as well. So you could actually have an alternative. You don't have to use Protractor, you can use WebDriver, or if you want to do so, you could implement your own integration layer. You could use Puppeteer or Test Cafe or anything else.

Nice. Yeah. And I think Protractor, from what I know of, is it hasn't been updated in quite a while. So is that still a technology you recommend people use using?

So for now, I think it's still probably the best version of WebDriver wrapper to use with Angular projects. But in the future, I Would like to explore other integrations. So like I said, like WebDriver, IO or Puppeteer and so on. So I'd like to give people alternatives so no matter whether the given tool is supported or not, they can make their own choice and use whatever works best with their project.

Great. Now, one controversial topic. Not controversial, but it was first ever Automation Guild in 2017. John Ferguson Smart joined us and I believe he went over the screenplay pattern. It was either the first or the second conference we had.

Right.

And so it's been a while and I still don't see the adoption of it within the community. I don't see a lot of people embracing it. Yeah, it seems like it has a lot of benefits. So I just want to get your idea first. You know, we've mentioned it a few times. I don't know how many people even know what it is. First, what is the screenplay pattern? And then maybe we could dive into the different parts of it and how it helps people create more maintainable, reusable automation. So I guess what is the screenplay pattern?

Sure thing. So many years ago, I think that was in the late 2000s, Anthony Marcano came up with this idea of expressing your UI focused tests using a pattern that would be an alternative to page objects. So the trouble he wanted to address with this pattern was that page object based tests, they tend to be quite tightly coupled to the UI itself. So whenever this UI changes, your tests have to change as well, and your tests often break because of failure or because of changes in the UI that shouldn't normally affect your tests. So that's the idea I had back then. Now we started actually working together around 2013 and as part of our work I started to evolve this pattern to actually support other interface of the system like rest APIs and so on. But what is this pattern? So the idea here is that you express your tests in a business language. So you introduce a concept of actors. So you capture the actors that interact with your system. So those could be the users, but also the external systems. Now those actors, what they do is they perform certain activities with your system. So those could be tasks or interactions. So things like clicking on buttons, entering values into form fields and so on. Now the core idea behind those activities is that you could actually express those lower level interactions using higher level tasks. So for example, entering a value into a form field like your username, and then entering another one like your password, could be expressed as a task to log in. So now a task now becomes a reusable component that you can use across your different tests. So now you no longer have to say that I enter this value into that field, I click on the button, I log in. So that's how you interact with the system. Now, another part of the scriptly pattern is how you retrieve the data from the system. So how do you take the data that you want to then assert on? So those would be the questions. So actors also have the ability to ask questions about the system, so they can retrieve things like the text value of an element on the screen or the response from the REST API and so on. So it's a nice way to model your tests in terms of the tasks and interactions that an actor performs against your system, and also the questions they ask about your system in order to verify the correctness of its state. So it's basically a nice DSL that helps you express your tests. So screenplay pattern is basically a nice domain specific language that helps you express your tests from the perspective of the user rather than from the perspective of the system being tested, as you would know, with page objects.

All right, so is it a replacement for page objects then? Or do you use both of them when you're writing an automation test?

So that's a very good question, Joe. So there is a concept of page objects in the screenplay pattern, but the way I like to think about them is I like to think about them as lean page objects. So typically page objects would describe several things, right? So they would describe the structure of your page. So the locators you'd use to identify specific elements the page, they would also describe how you'd interact with those elements. So they have methods like login or add an item to a basket. And they have a third thing which helps you to retrieve the data from the system. Now, since in the screenplay pattern we already have tasks, interactions and questions, page objects are limited to just describing the selectors or locators that you use to locate specific elements on screen, which means that they become much, much, much smaller. So typically those page objects will be a couple of lines long.

So it may sound complicated to someone that's never heard or seen this before. What are some benefits of using the screenplay pattern? I know from my own usage, I haven't done a lot with it. It makes it really a lot more readable to me. The syntax, is that one of the benefits of using the screen pattern as well as it makes your test more readable?

Definitely, that's one of the many benefits. So one thing I haven't talked about yet is that the screen pattern is a user centered approach to writing those high quality automated acceptance tests. It steers you to effective way, an effective use of layers of abstraction so helps you tests help you help your tests capture the business vernacular and encourages good testing and software engineering habits. Now, instead of focusing those low level interface centric interactions like clicking on buttons, entering values into fields, you describe your test scenarios in a similar way you describe them to a human being, an actor in screenplay speak. So yes, in this way you make your tests much easier to read, much easier to comprehend. Every layer of those screenplay tasks focuses on a specific level of abstraction. So you could have a task to log in, but you could also have a task to add an item to a basket, you could have a task to verify whether the price is correct, the voucher has been applied, and so on. So basically you write simple, readable and highly reusable code that instructs those actors what activities to perform and what things to check. Now, domain specific test language you create is used to express those screenplays. So the activities for reactors to perform in a given test scenario.

Nice. So you do work with, I believe you work with a lot of different clients around the world. Have you seen anyone say before using the screenplay pattern and then after, and what were the results of that transformation?

So one thing I see is that test scenarios, they tend to use much less code if you go with the screenplay pattern, as opposed to the more traditional scripting or page objects approaches. Now this is particularly beneficial in those industries and those contexts that rely on testing workflows. So for example, if your application includes workflows, like if you're selling insurance, for instance, those insurance forms tend to be very long and very complicated. Or if you're an airline that sells tickets and you have duplicated workflow of choosing the seat and any additional service and so on. So the more complex the workflow, the more beneficial it is to actually express it using a screenplay scenario, because you can then capture the different bits of the activities your actor performs as reusable components that you can use in other scenarios as well. So like I say, one of the main benefits that you typically use much less code to express your scenarios, another benefit is that it's much easier for non testers to understand what's going on in this scenario. So for instance, you could use your test reports for audit purposes. So if you show a webdriver scenario, a typical webdriver test script, to an auditor, that probably won't understand what the test is doing from the business perspective. But now if you show them that in my Scenario there's a trader who books a specific trade on a specific currency pair and on, and he's only allowed to do so if this specific group of privileged users and so on. Then you're expressing the test scenario in business language. So now suddenly the audience of your test scenario has become much broader. So not only your tests become simpler, but also they're much easier to comprehend by people who are not necessarily from a testing background.

Now it's a great point. It seems like there's more upfront work, but once that upfront work is done, the benefits to me are unbelievable. And like you said, I used to work for a large company medical field that would get audited and having that report just say here's, here's evidence of what we did and they can read it just made it a whole lot easier for us to take care of things. And the report itself. Also, like you said, if you had a new hire, if you did the upfront work correctly, you could hand them almost the results of that test and they could read those, the reports and understand what's going on. How to perform it manually if they had to verify it is one of the things I saw as well.

Absolutely. And then if you combine Serenity JS with tools like Cucumber, you could also group your tests into, you know, capabilities and features to basically use your test documentation as a, almost as a user manual. So you're describing what your application is doing and how it's doing it and not only focusing on the specific low level interactions that a typical webdriver test would do, but to go back to your question about adoption. I think that's a very good question. I think, you know, changing the way we, I think using Screenplay Pattern requires you to change the way you think about your tests change. So move away from just simply verify what's on the screen to trying to understand what is the business purpose of running this test, what is the business value of this feature. And I think this might be quite difficult because you no longer in your comfortable test domain, you suddenly have to venture into the business domain and understand the purpose behind every single feature. But it also requires you to design your test. And I think it's a, if you're moving from a standard test script background, it might be a similar switch. If you had to switch from, let's say writing a UI using jQuery to writing it in angular or react, suddenly you have to think about creating little components and how those components would interact with one another rather than just simply listing all the low Level steps.

Absolutely. And I guess it also promotes the best practices of domain driven languages or domain driven type of development in that you really are thinking about the user. I guess. So that's critical. That is the whole point of testing. Right. It's not writing these cryptic automation languages, but actually writing as a user thinking, okay, as a user, what I need to do, what do they want to do? And being able to verify that.

Absolutely. And you know, the funny side effect of that that, that I've seen some time ago is when testers actually started to express those test scenarios, you know, using this cleanly pattern. And then they showed those reports to user experience guys. The user experience guys looked at the report and said, oh my God, that's ridiculous. How many clicks does the user have to perform in order to get to this specific part of the system? This workflow is terrible. We have to improve it. So the test has actually affected and influenced how the user journey was implemented in the system itself. So not simply verified. Yeah, no, this user journey worked fine. It actually influenced how the user journey was structured.

Yeah. It's interesting because it also could help you align your testers with your actual users. Once again, going to my other company as an example, they used to send testers to on site hospitals to see how radiologists actually interacted with their software. But almost, if you used your automation with the screenplay pattern, I think you'd have better understanding between the tester when they were there on site and the radiologist speaking the same language and going, oh, you can approve it here because you're using the same words, the same terminology and everyone's understanding what's going on.

Yes, yes, precisely. And very often when you try to capture those business tasks using the business language, you might find that you actually struggle because there is no good way to describe whatever the person is supposed to do in the screen. And then you can start asking questions. So, well, does this mean that no other way we implemented the user experience need to change?

So I'm just thinking, as a skeptical tester, a lot of them would say, well, the reason why people are using other patterns other than the page objects, because they're not using the page object correctly. Is that a valid concern or is the screenplay pattern really that different that it does do things that you could not do with the page object even if you did the page object perfectly?

Actually, I used to do page objects before I started working on the screenplay pattern. And one thing I found is that one of the many troubles with page objects is that they're not particularly composable. And one other thing they do, they also lock your thinking in a certain way. So for example, they make you think about your test from the perspective of the structure of your website. What this means is that you express your test as a bunch of page objects. Those page objects have certain actions on them. So suddenly you're think is focused much more on how my page is structured, how my interaction flow is structured, and you stop thinking about the user. So that's one of the main issues. But other thing that you couldn't do with page objects, even if you did them perfectly, is how you'd actually compose page objects. So for example, imagine a page object that represents login widget. For instance, this login widget now has a method to login and you have another widget that gives you an action to, let's say, search for a product. Now how would you express, how would you implement an action that actually involves both those two things? You wouldn't necessarily create a third page object that then invokes those two under the hood. I mean, if you did, then you suddenly coupled those three page objects together, you could implement those as separate functions and then interact with those two specific page objects. But then what you do is you'd slowly start to progress towards screen clicks, right, because you start to introduce additional layers of abstraction. So if you need to do this, then why not just switch to screenplay pattern altogether?

So Jan, I know some folks really were in early. They started using Serenity JS1. I know Serenity JS2 is the newest version. Is there anything people need to do in order to make the migration over from 1 to 2?

That's actually a very good question, Joe, and I'm glad you asked that. So there's a number of new features that Serentyjs 2.0 introduces, and there's this different module design. But how does this actually affect your tests? So one thing I tried to do with Serenetejs 2.0 is to maintain all the public APIs intact. So if you created your scenarios using tasks and interactions and code questions and so on, all those interfaces remain the same. So you wouldn't have to change your scenarios. But there's a couple of things you need to change around your configuration and around the node modules you'd use. But to help you with that, I actually wrote a guide to help you in this migration, step by step. So if you go to serenityjs.org There is a guide titled Upgrading to Serenity JS 2.0 that I read. And this should help you migrate your tests to the new version.

Okay, before we go, is there one piece of actual advice you can give to someone to help them with their automation testing efforts? And what's the best way to find or contact you?

I think as testers and as developers, it's important for us to be open to new design patterns, new tools that come out, and to stay up to date with how our industry evolves. So what I would like to suggest to people who perhaps haven't used SerenityJS in the past is to head to SerenityJS.org website and try to follow the tutorial and see if the way I suggest you rewrite your test. So using the screenplay pattern helps you change the way you think about your tests? I know. If it does, perhaps that's something that you might want to try going forward. Now, the best way to contact me is you can go to my own personal website. So that's janmolak.com or you can find me on Twitter. My handle is yannmolak or LinkedIn. And if you'd like to ask any questions, if there's anything I haven't answered in this podcast, just feel free to reach out and chat. Oh, and one more thing and we've recently started a Gitter channel. So if any questions about Serenity JS or need any help with migration just now, head to Serenity JS GitHub channel and the community will be happy to help you out.

Thank you Jan for your automation awesomeness. For links to everything of value we covered in this episode, head on over to test guild.com a302 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is help you succeed with creating end to end full stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.