Reliable & Fast Synthetic E2E Monitoring with Tim Nolet & Hannes Lenke

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we're going to be talking with Tim and Hannes all about how playwright and puppeteer frameworks enable reliable and super fast synthetic monitoring and also ways to boost your end to end testing efforts. Tim is the co founder and CTO of Checkly with over 20 years of DevOps experience and Hannes is also a co founder at Checkly with a bunch of developer experience I believe. He previously founded the company Test Object, which is a mobile testing SaaS product that I also believe was recently acquired by Sauce Labs. So if you want to discover a way to unite active monitoring capabilities with reliable end to end tests, you don't want to miss this episode. Check it out. The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsorship action to try it for free for 14 days. Check it out.

Hey Hannes and Tim, welcome to the show.

Hi Joe, thanks for having us.

Awesome. Great to have you all. So I'm really excited to talk to you. Playwright has been a really hot topic I've been hearing more and more about, and usually it's Selenium you hear about. So I'm just curious to know before we really dive into it now, what's the deal with headless automation? I think you all also do something with the headless.dev site, which I think is all about headless automation as well.

Yeah, that's correct, Joe. So it's a long story. So if we're going to talk about Playwright, we should probably also talk about Puppeteer in the same breath. I guess. To be very short, we think, or I think specifically when I started Checkly, my company is that Playwright and Puppeteer both are automation frameworks to talk to browsers from JavaScript or Python. They are revolutionizing what you can do with browsers in code. It's not only testing, it's not only monitoring what we specifically do. There's a whole bunch more you can do and it's hard to answer it in one sentence like what is the revolution? Because there's so many different topics. So I guess we have enough to chat about. Faster is a term you heard here more often, more Stable, easier to use, more modern. I'm making air quotes, so a little bit hard to pinpoint what modern is. I think that's it in a nutshell.

Hannes, I don't know if you have anything to add. I've been a big believer in more headless type of automation. A lot of people, even the creator of Selenium told me if you have 1,000 selenium tests, you do an automation wrong. So any thoughts on that?

Yeah. So I mean what headless automation basically changes is first of all it's faster to execute a test, right? And it's easier. So headless is nothing more than executing your browser without an ui, which is essentially making it faster. And if you are able to execute scripts automation faster, then you're able to do a lot of sophisticated things in your e.g. development workflow without the need for longer breaks essentially. And this is just opening a whole space of use cases. So it's not only, not only Puppeteer and Playwright which you can execute Headless. Right. So there are also a lot of other frameworks and essentially that's one ingredient which is changing how also developers look at automation.

So I think another thing people struggle with as we go towards Headless is technical or the programming skills that are needed because it's not. Sometimes it's over complicated or it's JavaScript and sometimes you're not. People aren't familiar with JavaScript like Puppeteer and playwright, you kind of really need to know how to develop in JavaScript. So I've been seeing a lot of recorders coming out. So I believe you all also have a headless recorder. And if so, could you just tell us a little bit more what the headless recorder is all about?

Sure, maybe I should pick that one up. That was the first one I built. Build our current headless recorder offering first. A small correction, Joe. Puppeteer and Playwright are also drivable now from Python and I think from C also there's quite some development there. However, the main project is being developed Puppeteer by Google, Playwright by Microsoft as a JavaScript or actually TypeScript framework. And many of the use cases and the examples you will see are in JavaScript of course. But yeah, you need to still there is a little bit of, let's say. Well, I wouldn't call it a stumbling block. But yes, you need to be able to code to drive Puppeteer and Playwright. However, you can generate code and that's what our Atlas Recorder does. It's a Chrome extension, it's on the Chrome web store. It's on GitHub. It's an open source thing. We're doing pretty well. We have 12,000 GitHub stars. So that's a lot of virtual Internet money that we have there. And, and what we essentially do is you press record, you click on a bunch of stuff in your website, only works right now on website, not mobile. And then you click stop and then we generate that code for you. So the output of our recorder is just more code. This is kind of important because we do believe that that's what you want in the end. With the headless recorder, I didn't want to build a thing that hides it from you or that kind of, you know, makes a proprietary format that you then can execute it literally just it poops out JavaScript for you and sometimes you need to do a little bit of editing because the world isn't perfect. But yeah, we're doing quite well. We have some bigger plans for it to make it even greater and nicer and fulfill some other use cases.

So essentially it's a kickstart, right? It's kickstart for you to get started with your scripts and then you can take that and make that a larger automation suite which could test your websites or monitor your websites.

I assume this is open source, and if so, why invest in an open source project? I know you guys very busy, you also have a thriving business, so why funnel off resources to go to an open source type project?

So I mean, our users are JavaScript developers mainly, right? Or let's call them JavaScript experts. They need to have an easy way to get started, right? And the easier the way is, the better for us because then the users get to the success earlier. And this is why we're investing into open source and investing into the Headless Recorder. And also another initiative which we are driving is the headless dev, which is essentially a portal to teach you and to educate you on headless automation. Puppeteer and Playwright. We hope that these initiatives help the community to grow. And the community of Puppeteer and Playwright is a relatively young but fast growing community. And we hope that we can do something or edit here. Maybe we think that our investment will help the community to grow even faster. And this is important also for our business success.

In the end, I think that there's one thing I could add there is that the, the headless recorder that we have right now and the headless dev, which is this portal or knowledge base, they fulfill part of the life cycle when you adopt as a user a framework like Playwright or Puppeteer which we have strategically invested in, in our paid product. This is very core to what we do. So we are all in. But what you typically see is that there's someone who's interested in it, heard the name, wants to play around. So there's a part of the education that we need to do, that's the headless dev. Then the education part is over and people are really going to try it out, write their first code. That's when this, when our headless recorder helps you to write that first code. Hopefully we are doing a good job and we're nice people, we're a good business, they will find a use for our product. And then of course there's this. What are you going to do with those? I run a bunch of tests, or we call them checks, and then I have some output and something breaks or something goes right and you want to look at what the results are. That's kind of the next step that we're working on right now. So we see the open source initiatives really good for, as Hannes mentioned already, for the onboarding of people that might later want to use our product. But also in general, they don't have to use our product at all. We benefit if the community grows, if the level of knowledge goes up in the wider community.

So you know, Hannes, I think you've mentioned developer or dev three or four times. A lot of times when people think of automation tools and testing, they go automatically a tester or qa. So do you actually see developers embracing these type of tools?

Yes. What we see in the market is that basically the more teams are picking up DevOps and modern DevOps that they are also responsible for the reliability of their application. And this means reliability in the development phase, but also on production. So they get called if something is wrong in their specific service, on their specific page. So there's a need for tools which fit into developer workflow. Traditional or testers who are working together with developers want to have, want to adapt tools which also their colleagues could understand and help with building out an automation suite in the end.

And I could make it more practical. Even when I started coding checkly roughly two years ago, I had some other jobs before that, of course, and these were jobs mostly at startups, small dev teams, maybe whole company, 40 people. We didn't have QA people. That was not the hiring list. We had engineers and engineers did everything. So it seems, and this is maybe anecdotal, but there's definitely a trend where it's just not something that smaller businesses or SMBs invest in right now. They expect developers to do it. You do the qa, you do the monitoring. This was really the underlying trigger for me to actually start building checkly and adopting that role that wears multiple hats. I'm the coder, I'm the tester, I'm the operator. In the end, I'm doing everything. And if these tools talk to each other and I can use the same language, the same frameworks and the same things that I know, less overhead for me, less context switching.

So I think another thing people struggle with is there's so many tools out there, so how do they know what to choose? So you know, you have Selenium, you have Puppeteer, you have Playwright. Is there any criteria you use or you give your customers to know, hey, you should use Playwright in this situation or you should use Puppeteer in this situation, or is it just up to personal preference?

That's a good one. Just to stick to the topic Playwright and Puppeteer, because I'm pretty sure Hannes has way more knowledge about Selenium, having worked at one of the bigger vendors that has that. At its core, Playwright is relatively new. It is, it's. No, I wouldn't call it the fork of Puppeteer, but many of the folks that worked on Puppeteer are now working on Playwright. So there's definitely a large overlap in the API, not so much in the actual code on the backend. So what you see happening right now is that Playwright is on a rocket ship. They are developing really fast and adding all kinds of, let's call it, higher level features that have to do with recording videos, recording test scenarios, stuff like that. This is all very recent. Some of that shipped like two weeks ago. What I see right now is that you can't go wrong with either. They have very similar features, but. And hopefully the people at Google will not look angry or will not get angry. I think the Playwright team is really nailing their feature roadmap. And I have been in situations already at our company, when there was a support ticket, something didn't work, a page didn't render correctly, and I said, why don't you flip from Puppeteer to Playwright? And the bug went away. It worked. That for me is a big signal. So the reliability went up by the. Basically, the rest we'll have to see. I think the coming months, six months are going to be very interesting on how both frameworks level up and maybe start, you know, going in different directions. That's possible. I don't know that yet because I don't have all the details from the companies building this. So we are actually quite in the middle of it. And there's other players also. Cypress is a big one, of course, they have their own thing. I am very curious what's going to happen there in the coming six to 12 months.

I can just add that Playwright is making more efforts to support cross browser which could be main difference for testers in the end who really want to make sure that their browser works on every different device out there.

That is a good one. I completely forgot that the big selling point when Playwright came out of the gates was hey, we support Firefox, WebKit, so the Safari framework and Chromium all at the same time. And I think they're almost completely at feature parity right now across the three big browser families.

Yeah, nice. So Hannes, I'm just curious to know from your experience then I think a lot of teams have a lot of investment in Selenium or UI tests. Do you see teams replacing Selenium with Puppeteer or Playwright tests? Or do you see like a mixture of maybe 10% selenium end to end, more front end type tests and then lower level, kind of faster headless type testing using languages or tools like Playwright and Puppeteer?

Yeah, good question. I mean what I've seen in the past is that especially newer projects, right, so projects with modern web applications start to look left and right and are really trying to understand what is the right framework here. And these are the frameworks which normally adopt newer technologies like Puppeteer and Playwright or Cypress, where when you have already invested into a lot of Selenium tests and they work out for you or other frameworks and this works out for you on a larger scale, why should you change automation? And coming up with a big automation suite is always an investment and you should always think about hey, does it make sense to throw that investment away and do again enough benefits of from building something from the scratch. It's like building software. This is a question I see teams asking and sometimes the answer is no, stick with what you have and just be happy with it.

So we talked a lot about the testing piece and I think we've already mentioned DevOps and monitoring and the whole life cycle. And so a big shift I've been seeing is people focusing in on the full pipeline for the software development lifecycle and a big shift going towards monitoring, utilizing their scripts to give them not only functional test information but also some sort of production or help type information. So you talked a little bit I think or in the pre show we talked a little bit about synthetic monitoring. So before we get into the monitoring piece, I guess for folks that may not know what is synthetic monitoring, so.

We talked about automation scripts, right? And synthetic monitoring is in the end, scheduled automation script. So something which runs on a schedule, maybe every minute, maybe every 10 minutes, you name it. In the end, synthetics enable you to have an artificial user in the end. So it's not a real user which is interacting with your application, but really something artificial. So a script which is simulating the interactions with your application on the production system. And if that goes wrong, then you might want to get alerted and developers might want to get a call to fix the issue. And this is what synthetics is in the end. The benefits of it is that if you have these artificial users, then they face an issue. It's not a problem for you as a company, real user might face an issue. Then you maybe lose a customer or lose a user, but maybe just lose trust of that customer. And this is why synthetics are important for users, for companies.

Yeah, correct. And maybe just a very practical example that I always tell customers and potential customers is, hey, you got a web app requires a login and you log in. Any SaaS tool has it. You log in your dashboard, loads, fills up with a bunch of stuff that needs to work, right? Yeah, that needs to work. Good. Then you create a script. Use whatever you want. We recommend playwright or puppeteer that does that action. Create a test user, something like that, but it's representative of your application. And that user or that script logs into your application, loads the whole screen, and then you write a bunch of assertions like, hey, this needs to be there, that needs to be there. It's basically an end to end test and run that every 10 minutes. What's interesting is that doing that gives you a whole bunch of information that your application is working. Because in most sophisticated applications, the screen that loads does a whole bunch of background calls to fetch data from services from your database, et cetera, et cetera. If you have a green check mark on a dashboard that tells me, yes, this page is working exactly as I want right now, and in 10 minutes, and in 10 minutes it gives you a lot of confidence because, well, we all know that unit tests and end to end testings while you run it before production, they involve mocking, they involve all manner of things where it's similar to real life, but it's not really real life. And you can get that last bit of reliability or trust that you just shipped a correct release by just having it monitored on a loop every 10 minutes. That's basically it in a nutshell.

Great. Now I am older, so I want to see if this is the same scenario. So back in like 2000, had tools called WinRunner. We had a way where we use something called like business process testing or business monitoring testing, where we'd install our scripts on machines and we have to physically ship them to our insurance agents and then they plug them in and those tests would run every 10 minutes. And then if you got like two red in a row, we'd get alerted and we'd know that there was something wrong with the website. Obviously technology has changed and things have become more effective. But is this the same type of principle?

This is exactly what we do, minus the shipping of the boxes.

Yeah, absolutely right.

It is the exact same thing.

Correct.

Cool, cool. So are you running that in different locations as well? Can you do that? And like, because you're able to use the cloud now, you can run in maybe Asia and run scripts in Europe and US and then be able to determine if you have issues in certain areas as well.

Yeah. I think one of the interesting bits, which we actually didn't discuss yet, but it has to do with these new frameworks is we leverage AWS Lambda and we can run wherever Amazon runs. So we have I think 20 locations now that just added Milan, but we added that too. So in Italy, South Africa was added recently, which is great. So we shipped it there. The interesting thing is that the workloads that we run are fairly lean. This is because of these frameworks, they're more resource efficient. So there's very, very little overhead for us as a small company to just distribute those workloads all over the globe. We give a nice UI to the user that can click their little country flag and they can run these from anywhere on the planet and you get interesting extra information. Not only does my app work, but how fast does it load? Are there any, you know, weird issues going on with proxies all over the world? And we do see quite some interesting bits where, you know, one application works from India, doesn't work from, I don't know, the US because I don't know, some part of the Internet in between broke? That happens quite, actually, yeah, 100%.

So, you know, I know you have a tool, Checkly, we talked a little bit about it and obviously this is the functionality, it seems like that it handles. So can you just give us like a high level overview of what a lifecycle would look like if someone was using Checkly? Because you did mention lifecycle testing is important. So I'm just curious to know like end to end using your solution, like how do we unite Aerithin and what does it look like and what's the output?

Checkly does two things. We do synthetic monitoring using Puppeteer and Playwright and we do API monitoring because some applications are APIs, they don't have anything in a browser. So one of the, that's kind of, that's the core of Checkly because you want to have both next to each other. Weirdly enough, this is sometimes overlooked or it's an add on to an existing API platform or the other way around, there's a synthetic monitoring platform, you can add a little bit of API. We just treat them as exactly the same. They are two core tenants. So what happens is, well, you log in, you create a new check. That's what we call it, hence Checkly. If you want to do a browser check, as we call it. So render something in Chrome and assert it. We give you an editor, code editor and a whole bunch of examples and of course this recorder and all the educational material and you code your first check. This could be literally as simple as go to my homepage and use an assertion library and we just use the open source ones that are out there that are very, very well known and assert that the title is correct, there's a specific name in the screen or something like that. This could be five lines of code. You press save and rerun it for you every 10 minutes. We can give you a whole bunch of fancy options, but that's the core of it. The last step that every customer of us does is when it breaks, where should we alert you? So that's where the monitoring part comes in. And we have basically all the options that you would expect there. Email, SMS, Slack. It's like PagerDuty and OpsGenie are services we integrate with. So typically for the on call service, you want to get a message there when things break. You can add to this, the API part. This is very similar to if I show you a screenshot, people would recognize it. Oh, that's one of those API request builders like Postman or Insomnia where you put in a URL, you can edit all the headers, you can add authentication, all that kind of stuff. And what you end up with in the end is very nice looking dashboard that shows you all the key health data of your application. And APIs, we can check every minute for you. The browser sessions that we spin up, Playwright, we run them every 10 minutes and for some customers, actually every five minutes. I'm lying because that's probably more than enough. And that's our starting point. We have a whole line of extra things if you want to take it to the next level. This comes down to a lot of extra scripting you can do with your own code and actually triggering these suits of tests, as we can call them actually every time you deploy something. So lots of Our customers use GitHub or another service and they deploy to, I don't know, to AWS or Heroku or services like Netlify or Vercel. And what we allow you to do is not only run these things every 10 minutes or every one minute, but trigger them explicitly every time you do a new deploy. So this is kind of like an end to end test that comes right after your deployment and hopefully also catches a lot of bugs before they really start annoying your customers.

So I guess one objection or one thing someone could ask is a lot of times because you're in production, it may just be a superficial test, but a lot of the business value might be in someone adding to say an insurance policy. And obviously you're not going to delete a policy or do destructive type activities in production. How do you handle that? So you're not just getting high level information, but you're actually getting at the critical areas of your business. Does this tool help you with that or are there different techniques you can use in order to get around that?

Yeah, so this is a very good question. So on the what we call browser checks or an end to end test with Puppeteer or Play, right? You have basically all the power there. You can say like, hey, let's create an insurance policy in this script. Do a bunch of checks like, okay, yeah, this should be there, that should be there. And then delete it again in exactly the same script. So you can do a little setup and a tear down. That's pretty powerful already. Interestingly, for the API part, we did exactly the same. We allow you to run a script before the API check and after the API check. So we follow the exact same pattern that you see with traditional testing or end to end testing. Setups and teardowns that create a well known situation which you can act upon and then remove that. We do that ourselves with our APIs and lots of our customers are using that because, well, it's, it's kind of the holy grail of this testing in production, if we can call it that. Two remarks though. Use a test user, do not use an actual customer. That's bad or some admin user that's not good. So we advise you to always create a test account and keep it simple. I've seen some users and I see that in the wild too. They go overboard, they do too much. That last 5 to 10% is maybe not necessary. Keep your scripts as tight as you can, keep them as you can get them because they will get more complicated and the Internet can be flaky and there might be some glitch and then you need to add more error handling inside of your test script. It can get bloated very quickly. So those are always the two tips. Use a test user, keep it as.

Short as possible and if all this doesn't work, luckily these framework have some capabilities to mock behavior. So there's a possibility to intercept quests and make sure that you're mocking your website on the fly.

How long have you been around? I think two years. So do you have any feedback on how this has helped companies like Real world? Maybe a case study or things you may have found? They said, oh wow, you know, it actually implemented it here and we've got X amount of, I don't know, whatever.

So I think there's two cases. Vercel, formerly named Zeit, they rebranded this year. They're an early customer of us. They use it for every deployment that they do of their front end. They use it quite extensively. They have hundreds of checks running customer sites, specific API endpoints, but also using this triggered thing. Like every deploy they see their results in GitHub because we integrate with that. So in a pull request you get a nice little green check mark. Everything is correct and if it's not correct, we show you what's wrong. So that's a pretty massive use case they have there. Then there's another company which really, they have a really interesting product. They're in the security space and they blew my mind. What they did was the following. They have some software that runs on premise which they have no control over. It's one of their products, they can't really get to it. One of their customers installs it and then if something breaks, they're like, yeah, well, you know, we can't really do anything about it. But they do have access to a health endpoint. So what they did was they integrated our product into their rollout. So one of their customers their product through our API. Because we have an API for everything through our API a check gets created and if something fails, a webhook gets fired because we have webhooks and that webhook calls their system and that system then sends a nice branded message to their customer like, hey, it seems to be there's something wrong with that installation that we don't have access to. You might want to check this and this and this. So that was, for me, a completely amazing use case of how they automated through our APIs and our webhooks their reliability process for that specific product. Pretty cool.

So, you know, you also, once again, you've mentioned reliability a few times. Is this because there's been, you've seen a push towards site reliability engineering and do you see this tool falling into that type of silo or like, who would be the main person that would utilize this tool or put it in place?

So the main push which we are seeing is, as I mentioned before, that developers and development teams are now responsible for the reliability of their site. We mention reliability quite often because of basically two things. First of all, monitoring has to be reliable. So you don't want to wake up in the middle of the night because of a flaky check somewhere. So the underlying checks and scripts and frameworks and technologies have to be very reliable across all of our customers. And the second thing is reliability for digital products becomes more and more important. Customers do expect reliable products and our customers understand that. They understand that their website, their digital products have to be reliable all the time. So, yes, there is a shift towards reliability, maybe towards sie, but even more a large shift that the expectation of customers and of our users. Yeah, it's changed.

So we touched on a lot here. You're our newer tool. We touch on pretty much a lot of the buzzwords also that are out there. But one thing we didn't talk about is AI. A lot of times people slap on AI with a new product. I don't think I saw that on yours. Any thoughts around AI or like, do you think you could see that playing a role anywhere in the functionality on your roadmap if it's not already there?

Yeah, totally. This is a scoop. We're going full AI. Everything's a.

I think it neatly segs into what Hannes just mentioned. We are in at core. We're selling reliability, we're selling trustworthiness. I can trust my process, my apps because I'm using checkly and they have my back. There's a big reason where we're not using AI or maybe machine learning, which could be actually a little bit closer. There's some interesting players that are using machine learning to update test code. Basically, that seems like an Interesting thing. I don't know anyone that uses it, by the way. But still, I'm not dissing it at all. I honestly think I see no real usage for AI. I also don't know a lot of stuff. I don't know a lot about it, but there's one on the machine learning side that's very interesting. Screenshot Diffing is a hot thing, I think. What are they called, that company with the nice hedgehog logo?

Percy.

Percy, yeah. They were recently snapped up by browser stack. I don't know if Percy uses machine learning. I have no idea. But they do screenshot diffing. So looking at screenshots, see if there's a change. That is a very good use case for something like machine learning, where how do you know that that difference in that screenshot is actually meaningful? That seems pretty fair. And I think we're going to see some interesting things there. But for now we don't have any big plans in that direction. So, Hannes, or did I miss our board meeting that we're doing so well?

You're right. So the important piece here is we're integrating open frameworks, Open Puppeteer, open Playwright, and there's always the possibility that someone is implementing AI or machine learning, which helps our users, and then the likelihood of that is really helping our users. Us integrating is totally there.

Okay, Hans and Tim, before we go, is there one piece of actual advice you can give to someone to help them with their automation testing efforts? And what's the best way to find contact you or learn more about Checkly?

So my advice is please check out our open source projects. One is the Headless dev and just read about Puppeteer and Playwright and maybe read the basic tutorials there. Maybe also check out the Headless Recorder and see if that Chrome extension is helping you to create the first scripts. And yeah, maybe also have a look at checkly. So checklyhq.com that's how you can reach us. We're also heavy Twitter users and using Intercom, so it's easy to reach us everywhere.

That is correct. I have nothing to add to this. Headless dev is, by the way, theheadless.dev. we got a Google dev domain.

It's pretty cool.

Thanks again for your automation awesomeness. The Linux evident value we covered in this episode. Head on over to testguild.com a330 and while you're there, make sure to click on the Try for free Today link under the exclusive sponsor section to learn all about Sauce Lab's awesome products. And services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.