Freeze Bugs in Time using Replay.io with Filip Hric

Foreign.

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating automation awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast and today we'll be talking with Philip all about Replay IO, what it is, how it helps you along with Cypress. He is a Cypress expert. I can't believe Philip has never been on the podcast before. I've known him for a while, but he is a automation guru. He's currently a devrel at Replay IO and he teaches testers about web development and developers all about testing. He also is a Cypress IE ambassador and he leads a Learn Cypress IO community on Discord and has a blog where he publishes Cypress IO Tips and you'll be able to find all these links in the show notes for this episode. Philip also is an international keynote speaker. I've spoken with him multiple times. Unlike applitools Future of Testing events, he speaks all over the world as well and he's a leading expert on test automation in Cypress IO and is an author instructor of Live Cypress Workshops and he has taught hundreds of testers and developers about good practices and advanced concepts for testing in Cypress. So if you want to learn about some good practices and advanced concepts in testing, what the heck is Replay IO? What's this new gig he has? You don't want to miss this episode. Check it out.

This episode of the Test Guild Automation Podcast is sponsored by the Test Guild. Test Guild offers amazing partnership plans that cater to your brand awareness, lead generation and thought leadership goals. Get your products and services in front of your ideal target audience. Our satisfied clients rave about the results they've seen from partnering with us, from boosted event attendance to impressive roi. Visit our website and let's talk about how Test Guild can take your brand to the next level. Head on over to test guild.info and let's talk.

Hey Philip, welcome finally to the Guild.

Thank you. Thank you very much and for the lovely introduction putting it all together. It sounds super awesome.

Well, it's really impressive and that's one of the things I want to dive into because I really like how your journey has progressed. You've gotten better and better and it seems like you're reaching higher and higher goals. The first thing I notice is you have a new gig at Replay IO as a developer Relations. I could be right. I thought you started more as like an automation engineer or SDAT at like slido. And then I think you went solo. Like I said, I could be wrong, but we'll find out. And then you have the Snoop gig. So maybe tell me a little bit about your progression and like what made you end up at Replay IO as a developer relations expert.

Yeah, so you got the journey totally right. So my first full time gig was at Slido, amazing Slovak company that I think it was really amazing place for me to work and to learn because I entered there basically with like close to zero knowledge about the whole tech world. And that's where I started with testing. And I know like some people maybe not don't like hearing it much when someone says that testing is like the entry gate to testing, but it has definitely been. Been for me. And yeah, and slido as a company allowed me to learn and to grow, which was also kudos to them. So that's where I started and alongside I started to explore Cypress. I started learning about Cypress and then one of my colleagues said, hey, you already know enough about this that you could teach someone. I was like, oh, I guess I could. So I went to a conference, did my first workshop, and then I gathered some courage to do the workshop on my own. And that's where we started meeting each other for the first time on the applitools events and I started my blog. So that's how I progressed. And at the beginning of the year I left Slido and I went for a solo journey to do the workshop thing on my own and consultations and stuff like that. So I've been on that for a certain period of time. It was fun, it was a lot of hassle. It was kind of crazy that I. I did my own marketing, I did my own homepage, I created my videos, I did, I did quite a lot. There's a lot if you want to go solo, but it is, it was really, really rewarding. And yeah. But little time before that, met Andrew Knight, who was DevRel at Applitools shout out to Andy, a great friend of mine, and he visited last year for. Well, he was close by, so I invited him to stay over and it was right in this very room when we talked about what is it that I actually do, where am I progressing? And we found out that we created together that tagline of mine, teaching developers about testing and testers about web development. And I have been feeling that this is the journey I am on and this is my mission, because I never knew which bucket I fall in, whether I'm more of a tester or more of a developer, somewhere in between. I don't feel like a developer, but I also maybe not always feel like a tester. So that is my sweet spot. And yeah, Replay I.O. was looking for someone like me who's doing exactly that. So that brings us to the present day.

Love it. And I love, like, you know, the way I grow is I watch people that are ahead of me and I see what they're doing and I try to emulate it. So I think if people listening to this, you're a great person to emulate because you started off, like you said, you didn't really know testing, you learned testing. And, you know, I don't know if you suffer from, like, imposter syndrome and things like that, but it seems like you got over whatever hump may have been. Like, I'm not a tester, I'm not a developer. But you taught yourself, and then you started creating content around a topic. And then I probably. I could be wrong. I assume replay I.O. found you because of all the content you were creating. Is that true?

Yeah, probably. They reached out to me because they have started this new product or this new area of where Replay could be used, and that's test suites. So they were looking for Cypress expertise, playwright expertise, and try to get a feedback on whether the tool they are developing is useful for someone like me as a tester. So, yeah, that kind of brought more and more conversations after that. Yeah.

So if not for you going on your own and doing it or sharing your wisdom, I mean, probably you wouldn't have had that connection. So I think, especially in this economy, what you did is great and people should do more of.

So sorry to jump into that, but there is a little thing that I would like to maybe say now that we are public and people might be listening.

Sure, yeah.

Because as we started, you know, like, put all of those successes together, it sounds super human. Like, and the journey that I went through, I tried to live by. One principle in terms of, like, the content creation, is that I always wrote a blog about something that I was currently learning. Something it didn't have to be, like, long form, super amazing stuff. It was just this tiny thing that I learned, whether it was that day or that week, because I knew that if I found a blog post like this one two weeks ago, I would have learned that two weeks ago and not today. So to anyone out there who's considering to write a blog, whether on medium or on their own, I really encourage you. Just take a thing you didn't know two weeks ago. And write about that it's going to help someone because you're on a journey, some people are in front of you, some people are behind you, and you're going to help those that need the information that you know.

Absolutely. And that's actually how I started. I started a blog, JoeCollinschow.com talking about Windrunner and Loadrunner and QTP. There was no blogs back then on that stuff and then just snowballed to what I'm doing now. So great advice. So not only are you helping yourself because as you teach, you learn yourself, but you also. It's almost like a little breadcrumbs you're throwing out that you never know what it's going to lead to. It always, for me, it always led to something better and better along the way.

Yeah, exactly.

All right, so I didn't know Andy Knight was also a therapist. I need to talk to him next time to help tone in my messaging. That's awesome. So what made you know Andy was doing a dev rel position? I guess at that time he's no longer doing it. But does that help open your eye to say, hey, maybe I want to be a developer relations expert as well? Or is it just coincidence that all of a sudden you had another company looking for a role like Andy had?

I actually knew for a while that I would love to land a Devrel job because, I mean, a couple of years ago I didn't even know what Devrel is and that it is a thing. And, well, Devrel is sort of. The area is very loosely defined. Right. But what I did and what I enjoyed doing was creating content about Cypress, sharing videos, talking on conferences, talking to people about their experience, help them troubleshoot. This was part of what I really enjoyed. And I didn't know that this could be a job. Right. And Devrel, at least part of that dev rel job description is doing this. So I felt like, oh, if anyone would hire me for a dev rel. Well, not anyone, but if there was an opportunity, I would gladly take it because it seems like something that I could maybe be good at. So. Yeah, but as I mentioned, not anyone, actually, I would like to move in an area that I really enjoy or maybe talk about product that I really enjoy. That's what I've been doing so far. And that's why Replay IO was very like, natural transition for me because I was already creating content, going to conferences, doing stuff like that. And Replay sort of fell into that space where I could, oh, this is a really cool product. I will be able to talk about it and like not lie about anything or not say it's super, super awesome and actually is. So yeah, nice.

So like you mentioned, maybe a lot of people don't even know what a dev rel is. I know it's a position that's been out for a few years now. I mean it's been around forever but it seems like it's growing in popularity. So what is a dev rel if someone's like, oh wait a minute, this might be something I like as well.

Well, for past couple of weeks I probably given like 20 different definitions, you know, like to my parents and to my wife and friends. So I tried to do one right now and it's probably going to be different from those 20 that came in before. But the way I understand devrel is someone who's at dev rel position is a public figure of that company, is trying to do a little evangelism about the product that the company creates. Which may sound like marketing but as we know marketing doesn't really work with developers. Right. You cannot just put out an ad and hope someone clicks on it. And the answer basically is in the name of the position developer relations. So what I want to do is to create good relationships with testers and developers, go on conferences, provide useful content which will be not always be about Replay but maybe about a broader topic like end to end testing. End to end testing, flakiness, Cypress, playwright, other frameworks as well. So that's probably basically going over the definition because I'm just defining it by explaining what everything I would do. But that's kind of how I did that for my friends and for for my parents, et cetera.

Nice. So is it meeting your expectations and what you thought it was and now what you are doing? It's in line with what you were hoping it would be?

Yeah. So I've been there four weeks. I think this is my fifth week at the time of the recording. So far it's great. I work with team of amazing world class engineers. We talked about imposter syndrome. Right. They're just amazing, super smart people. So I love talking to them. Currently I'm working on a new documentation so I need to think very deeply about what steps people need to take in order to get a good understanding of what Replay IO does. And that's been always sort of at the center of when I was writing a blog, when I was preparing a talk for a conference. So far it is, I'm at my sweet spot so.

So I guess the next logical question is, what is Replay IO? Is it a newer tool? Because I just started hearing about it from you, watching your tweets and your blogs and things about it.

Yeah, that is something that I also need to learn how to convey, like how to put that message forward. Because Replay IO is a tool that is sort of in a category of its own. So if I were to go with maybe let's do this with a couple of basic steps, right? So the most basic definition of Replay would be that it's a browser, right? But it is a browser with special capabilities. So normally when you run a browser, you open a page, you browse everything. You wouldn't use Replay like this. You would use it for your development or your testing workflow. Because Replay, the browser, has extra capability. It has the capability to record your application or record the webpage or the application that you are visiting. And so the recording that it creates, it's. First of all, it's video. That's fairly understandable. Second of all, it's something like a trace viewer in Playwright or the Timeline or the test replay that Cypress has. And on top of these two, you actually get the application runtime. So not only it records, like, DOM snapshots and network calls and console logs, et cetera, but it actually follows the flow of the information of the application. So let's imagine. I'll give you an example, right? Let's imagine you have a application and you're debugging that application. Not sure what's happening. Something. Something's wrong. So you go to your VS code or some other editor and you add a console log. To understand what's going on here, try to see, like, how is my application actually processing that data With Replay, what you can do when you create that recording of your application, you can actually add the console log after. So you first record everything. Replay is going to capture everything, and then you will go to your code and add a console log after. But you don't do that in VS code. You do that obviously, in the Replay browser. So to sum up, basically, the experience with Replay will have two parts. First one is creating that recording. And second part is like all of the dev tools that you might have in Chrome or Firefox, and then have those tools interact with the code that's been recorded.

Okay.

Does that make sense?

It does a little. How does that help, though? Someone that's using it, like, what's like. It sounds almost like telemetry of observability to me. As a layman Here, where it's able to, I guess, if there's issues, is it able to bubble it up? Because it has that tie into the application. So rather than having all these things you look at, it's like it kind of, I would assume, condenses it all into one location.

Yeah. So the way that you might use Replay, there's actually like two big use cases for us right now. I mean, there are more, but the two are like most prominent. The first one, imagine you are a tester and you want to replicate a bug, right? So you try to replicate something and then you set out those steps, send them to developer, and developer says, oh, sorry, I cannot replicate this. It probably doesn't exist. Right. You imagine the bug and the tester gets frustrated. Well, I gave you exact steps. What the hell is happening? It did happen to me. Well, if you were to find those steps using Replay, so if you open instead of Chrome, you opened Replay browser, you would record that bug. And if you were to send that recording to your developer, they will not only have like the video and DOM snapshots of what was happening, but they will actually have the code. So the source code of that recording. And in addition to seeing that source code, they could add a print statement or a console log to see. All right, so here's something that didn't match up, right? There was some timing issues. Something happened here. I guess I'll add a print statement to see what the information was like, what this function returned, what went into this function. And this can reveal some timing issues and other kinds of problems that applications today have. Another use case for that, since you have that concept of having these robust recordings which you can work with. Another great example of how it can be used is that since Replay is a browser, you can use it for your end to end tests. So you can hook it up to Playwright or Cypress or webdriver I O and use it as the browser of your choice. And those recordings would get recorded manually or not manually, automatically. Right. And when you have a flaky test, right, you have one test failed, one test passed. It's the same test. You can do a comparison and see what went wrong. And not only see what went wrong, but actually examine what was happening in your application. If I were to put it maybe back up a little bit on how to understand this all is that when we have like a front. When we have an application, we have usually like the front end code, which is like React, some kind of JavaScript, CSS whatnot, then we have the backend. So The API which, which we are communicating with, and then we have the data layer. Right. So your application is just like a bunch of functions. They will take the data, maybe from that backend, and then process it through the application. With Replay, you can see that data and how it is processed and to understand, like how your application actually worked with the data that was flowing through. So if you have a situation where you have function A and function B and function B actually uses information from function A, but in some cases you might get into a situation where they don't trigger in the right order, you might be able find that problem with replay quite easily because you'll console log out information from function A, console log out information from function B and see. Oh, okay. There was a timing issue and we actually have a flaky app.

So I'm just trying to. Back in the day, I was a performance engineer and I had a tool called LoadRunner and had a plugin. I forgot the name of the plugin where, you know, if you ran a test for a performance test and you saw a degradation or performance spiked up, it allowed you to drill down and it can even tell you like this SQL statement. When you ran this test, this is the SQL statement that took X amount of time to run. So you could tell. All right. Rather than just guess, you know, because it has. It's able to read the code, it was able to point to the exact SQL statement that was causing the slowdown. Sounds something like that. Does that sound correct? Kind of what it's doing.

I think there might be some similarities. Yeah.

All right, cool, cool. So how much time would this save someone that if someone's listening to, like, have you seen or heard, like, wow, you know, without this tool, I would never have been able to found this flakiness. Because flaky is notoriously hard to debug.

Yeah, I don't have the numbers yet, but recently I made a poll on a plugin and asked testers, do you spend bigger portion of your day debugging your test or writing tests? And the responses said that 61% actually spend more like larger part of their day debugging. So the potential is definitely there. And we've been working with some clients or potential clients right now, and they are running their Cypress tests. They definitely have some flakiness, as you would with any tool. And we've been able to drill down like without even knowing too much about their code. But we were able to drill down different sources of flakiness. Like there was an A B marketing campaign going on and that through of the tests there were some false positives that actually resulted in flaky tests. There were some timing issues, some modal window popped up too soon and all kinds of different sources of flakiness. And we could see those happening inside the code and we could do that in just like couple of minutes because we could easily narrow down like the moment where we need to look and, and also the functions and the components that are inside the application to see what they were doing, how they actually processed the data, what went in, what went out, and we were able to pinpoint here exactly is the problem. So this is what needs to be fixed. This sort of reminds me of another poll I did recently because I asked a very theoretical question. If, say you have non flaky app and non flaky server, what do you think that would do to your tests? Make them less flaky, more flaky or same as before, now 79% would tell it would be less flaky. Which is interesting because we are talking a lot about flaky tests, but we don't really talk too much about flaky app. And yet the result of the poll sort of indicates that we should probably be taking the source code, the application code into the equation. And, and that's what replayio does because when you have a trace viewer in Playwright or the timeline in Cypress, you'll get insight into what the test was doing, but don't really get too much insight into what the code of the application was doing. And you get that with Replay hard.

Is this to configure then like what do you need in order for this to work? Is it just the browser? I assume you have to have some sort of sniffer or tie in to the code itself.

Well, the integration is pretty simple. Like if you know Cypress right, you run it with NPX Cypress run and then that would choose the default Electron browser. If I could call it that. If you were to do NPX Cypress run browser and then choose Replay Chromium, which is like the Chromium browser with replay capabilities, then that would be basically it. In principle, if I were to dive into the code, there is say like a plugin installation, you need add something into support file, then also to your plugins file. But the basic principle is like, yeah, just use this browser and you'll get the recording.

So I'm just assuming because it's a browser, if someone's doing like if you had a bunch of testers and we want to just explore a particular part of the application that isn't tested really well. You want to do some exploratory testing sessions. Would you be able to use Replay IO to help you with that as well, or do you need to use an automated tool?

Yeah, you will be able to use it as a standalone browser. So what you can do simply is to open this Replay browser, click on Start recording and click through your application. You'll reproduce your buck, for example, you'll stop the recording, and then what you can do is to send that recording over to your developer. They will open that recording and along the video and the comments you might add into that video, they will have the full source code as it was executed during your session. So there will be no more like, oh, it doesn't happen on my machine. Because now we have the log of everything that happened on your machine. Not only the actions and clicks and types, but the actual source code which can. Now we see and we can take a look into every function, what it has returned, what information was passed into it. So basically developer at that point will have everything they need in order to fix that issue.

Nice. So I'm looking at my notes and it sounds like this. We already explained, it's almost like little time machines that being used in your end to end testing. So Cypress Timeline, Playwright, Trace Viewer, all this is. It's like a, a point in time that you can go back and see what happened, have all the logs for, to help debug what happened during that. That point.

Yeah, exactly. But in addition to what you might call a session replay, which is what Cypress and Playwright does, in addition to the session replay that Cypress and Playwright has, you will have the source code as well and also like the source code execution. So yeah, every function, what it has returned, what it has been passed into, you can look into that.

All right, so another dumb question, or it's a browser so you can use it for exploratory testing. You mentioned Cypress and Playwright, I assume then you can use it for any automation tool that can invoke a browser then that can invoke Replay IO, right?

Yes, exactly. So you can. We also have support for WebDriver IO. There's experimental support for Jest. I'm not 100% sure about implementation of Selenium. I would need to take a look into into that, but I think in theory, yeah, you can. Since it's a browser, you could use it. But with tools like Cypress and Playwright, we actually have like an additional UI that you can use. So you don't only see the browser and the execution and the video from the test run, but you actually have a side panel and you can go through each and every command and jump to the point of the code, jump back into the point where, for example, the click was triggered or to the point where the typing started or something happened, some API call was made, et cetera.

Cool. So I also assume this would help. I don't know if there's still a division between testers and developers. It seemed like roles are kind of morphing together. But I assume if there is a team of dedicated testers and a team of dedicated developers, this almost help the communication, which tends to be the biggest hang up I've seen with a lot of teams. That's something you see as well.

Yeah, definitely, definitely. Because if you are a tester, if you want to file a bug reports now, you're basically relying on the execution steps. Right. At least that was the experience that I had. Right. If I had good execution steps and my developer friend could replicate the bug executing those same steps, that would be great. Right. But oftentimes it would happen that if they would go through the same path, they would not be able to find the same bug. Right. And this way with replay, you can record that and have the exact same steps and the exact same thing available to them. So I think in communication it's going to help quite a lot in teams. So, yeah, I definitely agree with that. Nice.

All right, so we're heading into almost 2024. We're in October when this will be released. Cyprus versus playwright in 2024, who do you think wins? Is there, Is it not even a battle? Or are any other tools you see potential for in the future that you think people need to know about?

I think it's definitely a battle. Right. Everyone wants to take a bigger piece of the cake. My prediction honestly is Playwright is going to continue on growing. And I feel that maybe the year 2024 will be sort of the year where we could be able to tell the future of what the future of those two frameworks will be. I think if the Playwright adoption is going to grow as fast or faster than it has last couple of years, then Cypress obviously needs to respond with something amazing. Right. And if they will have a good response, I think it still will be an interesting race. If they don't, it might in a couple of months or years, it might be the case that Playwright will have more users. Right now, I think Playwright has like a 1/4 or 1/5. I don't remember the numbers exactly of what Cyprus has. As we Know, the industry, like, moves a little bit slower with adoption of new tools. So I think next year will show what the upcoming years will be. If, if Playwright continues their rocket growth, which they seem they might be, I think Cyprus will be in for a huge competition.

Wow. Hot take for someone that loves Cyprus. All right, very honest.

I love Cyprus like nothing has changed on that part.

Honestly, I think it's a better tool. I enjoy working with that quite a lot. I've been using Playwright for a couple of months already and I've been slow. I'm learning slow. So that's why I prefer. Still prefer Cypress. And yeah, that's something to consider. Like, how fast are you able to debug and how fast are you able to write your test? Because we know that Playwright is faster than Cypress. Right. Objectively, it just executes faster. But that's not the execution part. That's actually the part of the day we don't do. Right. We just hit the script and it runs for ourselves. And yet we're like, as a community, we're quite fixated on that. I think the very important part is being able to write your tests fast and being able to debug them fast because they will be flaky, like every time. When we're talking about end to end testing, we will have that problem in the future. And the sooner we can fix those tests and find out what was the problem, the faster we will move and the better coverage we will have. The better trust in the tests that we write we will have, et cetera.

Wow, awesome tip. I definitely agree with you. Okay, Philip, before we go, is there one piece of actionable advice you can give to someone to help them with any of their automation testing efforts? And what's the best way to find contact you or learn more about Replay IO?

Yeah, so this podcast is called Test Guild, right? So testers are listening and I. My number one advice to all the testers is go write an app, try it for yourself. I think that experience is really eye opening and it's going to help you understand what's actually going on during the development. It will build maybe a bigger empathy towards our developer friends. But also, and I think, like, that's most important, it is going to give you quite a good context and it's going to make you a better tester because the decisions you make as a tester on what to automate, what to maybe skip, if those are informed by the knowledge of the code and how your application works, they are going to be more effective. So number one, Tip. Build an app. Try it out for yourself.

Love it. And the best way to find a contact you Philip if people hear this and like oh my gosh I need to follow this guy or I need to reach out to learn more about.

Replay IO yeah I'm on LinkedIn, I'm on Twitter. That's where I'm mostly active. Or you can also go and check out my homepage philip rietz.com that's where you find me. And if you're if you're interested in learning Cypress then you can go to my Discord which you will find link on my page. So Philip Ritscom Discord and if you want to learn about Replay then go to Replay IO. We also have a Discord channel so if you are eager to try it out or you stumble upon some problem and need help, I am hanging around in that discord so feel free to reach out.

Thanks again for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com a473 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space base or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to TestGuild Info and let's make it happen.