256: Cypress.io Hot Seat Questions with Gleb Bahmutov

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of Test Talks. Today we'll be talking all about the myths, both good and bad, around Cypress IO where Gleb Moptive Gleb works at Cypress IO and he's a JavaScript ninja, imaging processing expert and software quality fanatic. He also has a PhD so call him Dr. Gleb. He knows his stuff. This is taken from our Ask Us Anything session where we put Gleb on the hot seat as the guild asked him any question they had around Cypress IO. For those of you that missed out on Automation Guild, just so you know, any guild conference, because everything's recorded, you can actually go back if you missed it and purchase all the recordings, get instant access at any time. To do that, you just need to go to Guild conferences dot com. And also, you're one of the first to know that I'm planning to do a complete rebrand. So currently I have three main websites. I have joecolantonio.com, which is my blog. I have Test Talks, which is my podcast, and. And I have Guild Conferences. So the three completely websites, they're hard to maintain and people don't seem to know I do one or the other or all three. So I'm thinking of rebranding them all under the name Test Guild. So going forward I would have just test guild.com where you'd be able to get blog posts, podcasts and online conferences all dedicated to any type of end to end automation awesomeness. So that's not official yet, but I'm working on it.

Just wanted to give you a heads.

Up to get you kind of familiar with Test Guild. So all conferences going forward would be like Test Guild hashtag Automation or Test Guild hashtag Security Testing. So I don't have to have different logos for every single conference and different swag. Everything would be Test Guild. I'm also thinking of starting two other podcasts. So it would be Test Guild Automation would be one podcast, which would be what Test Talks is now. So Test Talks would become Test Guild, hashtag Automation. And then I have another podcast called Security Testing. So it'd be Test Guild hashtag Security Testing, which would be a separate podcast. So it just helps keeping everything consistent. I'm too small to have, you know, five to ten different brand names. I want everything under one name to really make everything clear.

So Just a heads up there.

Test Talks is sponsored by the fantastic folks at Sauce Labs, the cloud based automated testing platform that eliminates the need to maintain your own Selenium grid and test infrastructure. Try it for free today. Visit testtalks.com and click on the Sign up Now link under the homepage Sponsor section.

Big Lab.

Welcome to the Guild.

Hi Joe, how are you?

Great. It's awesome to have you. And there seems to be a lot of excitement behind Cypress and also a lot of misunderstandings of what exactly it is and how it compares to other tools. So I thought we'd just jump right into it. All right, first question according to you, what are the three features of Cypress over Selenium?

I just want to start with that. Whatever we've done with Cypress is because we look at Selenium and we all use Selenium and we have nothing but respect for Selenium webdriver and all other testing tools. So I will say that the three features of Cypress I think does differently is that it runs in a browser but allows it to interact with your code sub network stub methods, see the DOM updates in real time. Any tool that's not running in the browser next to your task cannot do that as well. So with Cypress it's an advantage. We also have built in retries. We thought that modern web, modern applications are constantly updating themselves, but they also are unpredictable. A network might take a long time, so you cannot just expect an element to appear, let's say 100 milliseconds. So Cypress commands are always have built in retrieve and will retry the following assertions to make sure that even if element takes a little bit longer to appear, the test passes and it's not flaky and doesn't need hardcoded delay. And also we do concentrate on easy installation. So with Cypress NPM install a single command, nothing else to install and you get all the screenshots, all the videos by default on every CI on every platform. So easy of installation is there. On other hand, Selenium has a couple of advantages too. For example, with Cypress you have to write your test in JavaScript or language that transpiles to JavaScript because only JavaScript really runs in a browser. With Selenium you can write those tests in any language. If you already have backend operations written in a different language, it actually makes sense to use Selenium because it runs on your operating system, so you can easily work with backend operations. And also Selenium supports cross browser testing and reciprocates it on the roadmap. So Selenium already has it. So you have advantages and disadvantages and you should evaluate both systems and see what makes sense for you.

That's a great answer. A lot of times whenever you talk about anything a tool other than Selenium, people think you're talking bad about Selenium. We are not talking bad about Selenium. You just options. More options, the better. Every team is different and one tool may be better for your team than another. Not because the tool is superior, but that's just how your tool, your team works with a particular tool. All right, we got that out of the way. Next question is. Let's go to it. It's from Harry. This has been a big topic in this conference so far. AI solutions. So can Cypress currently or is it planning to utilize any type of AI machine learning to help prevent flaky tests? With the number of tools using this on the market, it's a must for us.

I think you can kind of understand this question in many possible ways. I kind of mentioned that built in retries that prevents flake because of unpredictable nature of networks and storage. So there is that. That doesn't require ML or artificial intelligence. If you're talking about updating selectors and which selectors should be used when you run writing end to end tests, we have a best practices guide. You can use specific data attributes so you don't depend on dump structure or CSS classes. And at the same time, if you want to go full way to ML helping you maintain your test and update with selectors, we are talking to companies like Mable. Maybe something will come out of it. But we're also evaluating statistical methods that take the history of your test and retry the failing test automatically. So that with a single failing test out of 1000 doesn't stop the deal, but still doesn't get lost when it's retried. And you can go back and see which test sometimes fail and have to be retried and maybe with a time race condition that you should fix in your test. So we'll evaluate every possible tool to make your test less likely.

Next question is from Nathaniel. He wants to know because Cypress is interacting with the web app directly, is there a layer of code example risk above that that Selenium would test but Cypress is skipping example. Would there be a difference between telling the app that I clicked versus emulating a real user clicking through the browser?

Yes.

So right now cypress is completely JavaScript solution. That means synthetic event. It's creating an Event object in JavaScript and sending it or dispatching it to a button, for example, which might not necessarily work the same way as native events. We are adding native events. So the next thing for Cypress on our roadmap is actually supporting native events. And this will enable you to do things like file upload or file download or copy paste into your clipboard. You will still have synthetic events, but you also will have a native event for all events. So this will remove like tiny gaps like that at the same time. If we go back to what I've just shown with app actions, the point of app actions is not to stop testing the UI and stop clicking, it's to when you're testing a feature act, you go through the page and you click on a particular button when you're testing X, but when you're testing feature Y, you don't go for feature X again for the page. Instead you go directly set up your state for your test and then just test feature Y to avoid this duplication. But slow part of interacting with page. So you still cover all the features from page at least once. You just don't repeat yourself unnecessarily.

My next question is, can we use Cypress IE with a BDD tool like Cucumber?

Yes. So the syntax of Cypress is very different. It's kind of declarative. It takes getting used to.

People wrote BDD cucumber to Cypress Preprocessor, which you use as a plugin to Cypress test and in that case you can write your test in BDD cucumber syntax, but will be automatically translated into Cypress. It's not our official plugin, it's something that came from a community.

Next question. This one's from Gouda. She wants to know can Cypress do full screen image comparisons against baselines in a similar way that Galen Framework does with strict layout design requirements? It would be a huge help if these tests would run directly from dev code, not a separate test.

It's absolutely wonderful, wonderful question. And visual differing is a great technique to ensure that the page looks the same.

Okay, so Cypress takes screenshots. It's a built in. You can take a screenshot of a full page or a particular element or the whole thing, including the test Runner ui. We actually recommend you take screenshots of particular elements so that if it's actually different, you know which part of the page is different and don't compare. For example, dynamic parts all the time. But there is no built in image desync in Cypress. We give you screenshots, but it's up to you to compare. Luckily people have written plugins that actually do that and two companies, Percy IO and applitools have their own plugins for Cypress that do excellent image diffing. So you can use those companies plugins to actually have excellent visual tiffing with your Cypress test. We don't have to do anything, we just provide a platform.

Awesome. So club I heard Applique Tools. What was the name of the first vendor?

Percy IO.

What is the suggested migration strategy for page objects to App Action migration?

I would pick the test that takes the longest because that's a painful test. It's going through a lot of actions and they all take long time and I would see if maybe part of a test makes sense to be moved into App Action. For example, if you're logging in through the UI.

And that takes a long time, maybe you can log in by calling the app directly with username and a password and maybe next an API call. So to shorten that way. So my migration strategy would be to find the pain point, usually long running tasks and see what some parts of it are. Better to be a single fast app action call.

Next question's from Christina and Christina wants to know by using App Actions you are not touching the ui. How are we sure that the functional elements work? For example, that you're able to actually click on an element or type in a field? Isn't this prone to error? I think you touched on this a little bit already. Yes, yes.

You still touch the UI and you go through a page like a user if you're testing that feature. So my test that we're adding to those all went through the page like a user. But when I was testing routing and I needed a couple of to dos, I didn't add to do's again through a page. Instead I just called App Actions and I immediately went to testing the routing. So you still cover everything, you just don't repeat yourself. Instead you use App Actions to avoid going through the page again and again. So you still will cover everything.

Next question is from Harrison Lee. Harrison wants to know how do I choose? How do I use Cypress to test against some web app in different test environments, dev QA, staging or production with different base URLs.

Excellent question. So in Cypress, where a lot of config values like base URL and you can set that config in Cypress JSON configuration file or provide it as a command line argument or as environment variable or even programmatically you can change that so you can point at a different base URL by just specifying on command line or setting environment variable and that way you can test run the same test again localhost or against staging or even against production. We have a lot of examples of that in our documentation. So if you search for that, you'll.

Find it very cool. Okay, questions keep coming. Next one's from Chung. He wants to know since Cypress is On top of JavaScript Mocha, can it integrate well with WebDriver IO instead of migrating? Also can we use Node API test and NOC mock framework in Cypress?

So Cypress does include the Mocha JavaScript engine. That's what it uses to actually keep keep your test and control the before and after hook. But the architecture of Cypress is very different. You cannot integrate Mocha with WebDriver. It's more integrated with Cypress already. So unfortunately the first part is no, but Cypress already include a sign on JS Library. That means you can mark anything in JavaScript. It also includes network control so you can Mac spy and control anything that your application is sending to the server. So it's already included. So think of this as NAC or already built in with perfect ui. So the second part you don't have to do anything, it's already done for you.

Next one's from Flamingo. I assume that's a alias, maybe it's not. That's cool. Can Cypress IO be configured to run with different browsers to run tests in parallel?

We have a built in parallelization, but it's not parallelized on your machine. It's not like a single Cypress instance when you're running locally will spawn multiple browsers. It's not going to work. First of all, your machine will just spin out of control because Cypress is driving your app really really quickly in order for test to finish quickly. So your MacBook Pro or whatever you're using will actually just become really hard from CPU usage. So instead we allow you to parallelize your test and run multiple tests on multiple machines. So if you have a CI run we have a service operation service but will load balance all the specs and all the specs will be executed on multiple machines in parallel. But it's a different tool from running multiple browsers locally. So you know, consult our documentation and it's a paid feature unfortunately.

Okay Glep, next question is from Cap. They want to know very cool addition to the developers adding tests to their code. But as a QA automation engineer, the only access we have is to the pages and not the application source code. Do you agree that end to end Test with pages and page objects still has a place even with this technique.

Absolutely. I understand, I understand it perfectly. Sometimes you don't have access to the code, you cannot rewrite, you cannot add task selectors, you cannot expose the objects that you would like in your task. So absolutely very skilled and it's going to be for a long, long time, maybe forever. Place for full end to end test where you literally click on a button selected by text, by caption, by label and where you cannot take shortcuts and you still have to create some kind of helper method so that your tests are reusable, easy to maintain and so on. And with cypress it just JavaScript, if you like page object models, you can do that. If you just like creating small utility functions that you can call, you can do that. If you. We even have like custom commands where you can write additional Cypress commands, but also kind of do more stuff through the page as you want. So we are flexible and unfortunately sometimes we have to go through whole system through whole page just because we cannot change the application under that. Absolutely.

Next one is from stas. He wants to know, do I have to pay for the Cypress Dashboard service more than 500 privacy test recordings to have test parallelization and CI integration in Cypress, or can I get away with the free Cypress Test Runner in this case?

Fortunately, the test parallelization is a failed feature. That's how Cypress actually makes money. But Test Runner is open source mat license and we give it away for free, right? It's not feature limited. We work on that. Half of our small 18 person company is just dedicated to the test runner. So in order for us actually keep bat free and development going, we have to make money and we make it on private project. And parallelization is one of the features that we can charge for. But soon, if you have a truly open source project, that means nonprofit open source library or web application or framework, we are, we're going to release an open source license where you can get all the features of Cypress Dashboard for free. So that's a compromise.

Next question is, do you have a Java binding for sopress?

So unfortunately not. Ultimately all your test code has to run in a browser. If Java can be transpiled to JavaScript and unfortunately it cannot, then it cannot run in the browser. We apologize for that. We understand that JavaScript is not the language that everyone uses, even though it should be. But we've done a lot of work to make sure that our documentation and the learning curve and the errors that you get if you do something wrong are so, so helpful to non JavaScript developers. But even if you a backend developer, if you're just programming in Java, you still can still write effective end to end tests for the browser. And if you send me any kind of suggestion how we can make this that better, we would be really happy.

Very cool. Next question. Can Cypress do remote execution like Cypress Selenium Grid?

Not yet. So when you run in CI you can parallelize and load balance multiple machines, but each one will still be in its own browser right there in the container and will control it. It cannot control remote browser yet. We are in talks with all the companies that provide like device farm of browsers. So stay tuned. It's just our architecture where we have to close control and launch the browser is not conducive to actually doing that just yet out of a box. It will require technical work.

Next question then, is, is the concept of app action something native to JavaScript or Cypress or can it be achieved in just Java?

I think it's native to JavaScript and it's in the sense that your Web application, that's JavaScript that can only be JavaScript because it's running in the browser and the Browser only supports JavaScript. If you want to call that code directly, your code has to be JavaScript as well. Now WebDriver has, you know, evaluate JavaScript where you can kind of simulate, right? You can send a command that will load your JavaScript but it's not going to be the same. It's always going to be the same serialized HTTP post request going to your browser and then the browser is evaluating that. But who knows if the object is ready to actually receive that. When you run your test in browser In Cypress, your JavaScript is executing in the same loop in the same event engine as the application code. So everything is so much easier in JavaScript.

All right, any multiple browser support time frame and when that'll be available.

We're doing native events which is a prerequisite for multiple browsers. So in the next 2/4 I would.

Say so 7 says. What browsers does Cypress support as of now? Is it just Chrome?

It's just Chrome. We're concentrating on Firefox and AE11.

Are there any reporting APIs that Cypress integrates with?

So we have built in mock Reporters plus Junit plus. You can use like Mocha awesome and output several versions of your report and then combine. So literally we can generate all the reports. It's up to you to ingest them.

Will Cypress work with older ASP NET MVC application?

Anything that actually loads in the browser, you can test it. As long as it produces something your Chrome browser can load, it's going to work.

Is it possible to test Windows native desktop apps with Cypress by using combination with WinApp driver?

No, it's very different architecture with testing web apps.

What kind of reporting tools does Cypress support?

So you can generate xUnit, Junit, MOCA.

JSON report? Anything that Mocha can generate, we can generate additional ones.

So any last words of wisdom before we leave? Gleb?

No, and thank you so much for listening to me. And thank you so much Joe for inviting me. I appreciate it.

Thanks Gleb for your Cypress Automation awesomeness. For links to everything of value we covered in this episode, head on over to test talks.com256 and while you're there, make sure to click on that Sign.

Up for a free trial link under.

The exclusive Sponsor section to learn all about Sauce Labs awesome products and services. So that's it for this episode of Test Talks. I'm Joe and my mission is to help you succeed with creating end and automation awesomeness. As always, test everything and keep the good.

Cheers.

Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com for full show notes, amazing blog articles and other automation awesomeness.

SA.