AI For Playwright Tests with Todd McNeal

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating automation awesomeness.

Hey it's Joe and welcome to another episode of the Test Guild Automation Podcast and today we'll be talking with Todd McNeil all about supercharging your playwright test with AI and all other AI innovations and automation. You don't want to miss this. If you missed out on the last show we did with Todd, we got a lot of great insight around it. Todd is the co founder of Reflect, which is a test automation tool that can execute manual test cases using AI. We actually did a webinar on this as well that I'll have a link for in the show. Notes that you definitely need to check out after you listen to this. Todd and the Reflect team has also recently launched a really exciting new AI library for Playwright called Zero Step that I've covered on my new show and I was really excited to have Todd join us to share all about this and other innovations he's been working on over the past year that I think you're really going to be excited about. Todd is really passionate about making test automation easy to create and maintain and I am as well. And I think you definitely want to stick around all the way to the end to see how you can really supercharge your test with AI. You don't want to miss it. Check it out.

This episode of the Test Guild Automation Podcast is sponsored by the Test Guild. Test Guild offers amazing partnership plans that cater to your brand awareness, lead generation and thought leadership goals. Get your products and services in front of your ideal target audience. Our satisfied clients rave about the results they they've seen from partnering with us, from boosted event attendance to impressive roi. Visit our website and let's talk about how Test Guild can take your brand to the next level. Head on over to test guild.info and let's talk.

Hey Todd, welcome back to the Guild.

Thanks Joe, thanks for having me.

It's been crazy the past year for you. I think. You know, your webinar on generative AI this year I think was the best attended and most registered for webinar for the Test Guild. So really excited to have you join us. Why do you think? I think I know why. But why do you think Gen AI has been such a hot topic and why Your webinar in particular really seemed to really resonate with a Lot of people.

Well, I'm excited to hear that. I think it's just so new and so different than other technologies that I think just people are very curious about where it's going to go. It's evolving so quickly.

Absolutely. So speaking about evolving so quickly, I know you were on the show in July, I think, but it looks like you've already advanced way past what you've done in July. So maybe talk a little bit about maybe for the people that missed the last episode, what is Reflect? And then maybe we'll jump into some AI approaches and testing.

Yeah. So Reflect is a web based tool for building automated end to end tests and we launched about three or four years ago. We've been historically we're really focused on no code and low code automation. But once we saw ChatGPT launch, which was just about a year ago, we realized that this new AI technology would be something that would change a lot of how we build software and test software. So we, for the past year we've really been focused on AI enabled features. So when I spoke to you back in July, we had just launched our AI feature set which would allow you to basically write a prompt within Reflect and have that be translated into an action or assertion in your test. So the way you could imagine that is if you wanted to click on the first entry in the table, you could just say click on first entry in the table and it would do that. You don't need to conform to any sort of syntax. Since we've launched that, about 2/3 of our customers are now using that feature and even more of them are using that. We use that for self healing as well. So even more of them are taking advantage of that. So we've really decided to invest a lot of our engineering efforts in that. And so we've been branching out with these new functionality that. Yeah, excited to talk more about today.

Nice. So yeah, I love when you get something in the hands of your customer. I'm curious to know if you found anything new like, oh, I didn't know they were going to use it like that or it really worked the way I thought it would, or maybe it works differently but better. Did you get any feedback around that?

We did, yeah. So when we first launched it, we launched with the ability where the prompt would translate into a single action. So you would say click on the first search result or click on, you know, input your username in the username field. What we realize is that customers want to do more with prompts and basically what they want to do is what would appear in a manual test plan. So in a manual test plan you don't just have things to say. Click on this, insert this, you have assertions, you have test data that's inserted there. You have things which are, you know, multiple steps, like filling out a form, like fill out the form with values that would a user would enter. Right. It doesn't really matter to me what they are, but it should look realistic. So what we ended up doing is adding features iteratively as customers asked for that. So now the prompts can do things like fill out an entire form. It can, we have examples around like, you know, scheduling something on calendly, doing things in like very hard to automate ecosystems like Salesforce and SAP. So we've over the past six months we've really been iterating to handle all those use cases.

Nice. Love it. So I think it's also important to know the reason why I opened up like let people know about Reflect is, you know, there's a lot of tools out there and everyone does things differently. So I'm just curious to know for the people that haven't tried it yet, what's maybe unique about your approach to AI and testing and what your company is focusing on that maybe others aren't or what your special sauce is? Maybe.

Yeah. So I think what customers like a lot about Reflect is that it's easy to use. So especially if you're a tester who hasn't done much automation or if you've tried other tools like this and found them to be a little unwieldy, a little too much to learn, a little, not as user friendly as you would expect. Users are usually happy with the user experience of Reflect. The other thing that we've really focused on is accuracy too. So you know, when you think about tests, running tests, but also maintaining the tests, a lot of times what happens is you spend not only a lot of time creating it, but also maintaining it, things are breaking all the time. So our philosophy is that ultimately the best testing tools, whether you're writing code or not, the tests are only breaking when there's a bug or when your requirements have changed. If other than that it really ideally it shouldn't break. And so that's the ideal that we've been chasing for the past couple years and we're getting closer and closer to it. But yeah, the accuracy and the ease of use is usually the things people like the most.

Nice. Another thing I think I like about your tool is, is a lot of times when you go with the tool they sometimes force you into. Like, this is how you need to create something, and this is like a workflow that may be completely different than what your team is already used to. So I'm curious to know what you think of your AI solution or how you may feel, how it needs to plug into an existing user's workflow. Yeah, I don't know if that makes any sense, but any thoughts on that?

Yeah. So I think if you're learning a new tool, obviously any change to your tool set when you're testing can have a big impact into your existing workflow. Right. And so that means not only what you have to learn, but also, does it work with your existing tools? Does it work with your test case management tool? Does it work with your bug reporting? How do I get reports out of it to report on, you know, defect rate or things of that nature? And so we think it's really important that we fit into your workflow and don't require you to really change these other things that connect into how test automation is done. So one of the things that we released recently with Reflect is basically a Chrome extension that allows you to run Reflect tests within your test case management tool. So kind of going back to this concept of if you're writing prompts for test automation, those prompts actually end up looking like a manual test plan. With our Chrome extension, you can actually take your existing manual tests in your test case management tool and run it straight away as automation. And so that's an example of really fitting into your existing workflow, where if you're a functional tester, you're probably living in your test case management tool and JIRA or both. So that's where we fit into there.

All right. Because it's mostly audio that people are listening to, even the talking head on YouTube. Can you explain a little how that works? How does it create an automated test out of manual tests?

Yeah, so the way that it works is in your test case management tool, you may have hundreds, thousands or more test cases, and a subset of those test cases are things that you're running every time that you have a new release that you're testing. So that might be every week, every two weeks. If you're testing in Salesforce, you're probably testing the service pack, which is three times a year. So what this approach allows you to do is you would go into your test case tool, and if you're running a single test case or set of test cases, our extension adds a button, which is basically just a run button. You Click on that button and it takes your test case and basically treats each test step as a prompt. So if the first step is insert your username X, the second step is insert your password Y, and the third step is click on the login button, we'll execute those as individual steps, just like if you had written it as a prompt inside Reflect. And what's cool about that is that the data is synced back both ways. So when you go into Reflect, if you start running that test and you say, oh, well, actually my test case isn't as detailed enough to be able to run it as automation because it says like login instead of enter username, enter your password, whatever. It's very easy to change that in Reflect. And then when you save it, it gets automatically updated in your test case management tool.

Nice. So this sounds really cool. I just think sometimes people get scared, and we may have covered this in the last show, that if they go with the tool and then they start integrating with the test case management tool, if for some reason management goes, oh, we can't afford the tool anymore, something happens that they can't run the test anymore. It's called vendor lock. I'm not applying. That's what vendors do on purpose. But what would happen in that case if someone went with something like Reflect?

Well, that's true. I mean, any tool that you're using, there is some sort of, there is some level of lock in, right? Like it's very rare in my experience for someone to use some sort of automation tool, whether it's code based or not, and then be able to switch over to something else and just basically export it and switch it. Even if you're using something like Selenium or Playwright, most when people move to a different automation framework, they're going to rewrite it because there's different, you know, different best practices and different patterns. It's usually a good opportunity too to kind of clean up the things that you kind of accrued over time. For us though, I think the technical, not the technical debt, but the vendor lock in is less than other tools because at the end of the day it's really just your test cases. Right. And so when you're running tests from your test case management tool in Reflect, the edits that you're doing to make everything work correctly is really just clarifying the test, making the test more unambiguous, making it more specific in certain cases. And so that's just really kind of improving your test cases. You can kind of think of it as the AI Works best if you're writing the test cases so that if a new team member came in and was reading that test case without much context of your application, they'd be able to perform it and know if it passed or failed. That's the AI. So because there's not like a lot of coding involved or a lot of specific things that you're doing and reflect outside of just regular actions that would get synced back to the test case management tool, it does reduce lock in a bit.

Nice.

I know.

Another objection I know I'm going to hear a lot of, or I do hear a lot of is especially with low code, I don't know why is why would someone want to, say, run a functional test case as an automated test rather than roll their own and write their own automation script? I know a lot of people sometimes are skeptical of black boxes.

Yeah.

And because they don't know what's going on, they can't tweak the code. So any thoughts on that?

Yeah. That's been a criticism of low code and no code for a long time. I mean, low code and no code even predates Selenium, as you know. So a lot of people have faced issues where there's a limitation with the tool. It doesn't let me get my job done. It would be easier if I could just code it right. And I think there's a lot of legitimate concerns there. I think with AI, what's really exciting about it is it's next level accuracy. Yeah. Know the self healing from five years ago. It. It improves upon that a lot. It improves upon some of the frameworks that have been around a while, like Gherkin, where you really have to be specific in your syntax and underneath the covers. It's just code anyway. With AI, if the AI is accurate enough, it kind of solves for all these problems that people have been trying to build solutions for for years. And that's why it's so exciting. But yeah, I think people need to vet for themselves. Is the AI accurate enough? And we think it is and our customers are having a lot of success with it. But it's something I think teams would need to decide for themselves.

Absolutely. And I don't mean to keep coming back to this, but you did do a webinar and I know in the comments people like this looks unbelievable. And I heard after that some people actually tried it. I think you have like a free trial people can check out and they said it worked just like it did in the webinar. So have you heard that from customers or have you seen do like how much of if someone want to go with an AI solution, how much of a overhead is it to get it actually working the way a vendor may claim it works?

It depends. I mean I think it's a lot of it is similar to how you would think about a new automation moving to any automation framework. Right. Like what's going to make it, make that project more successful is if you've spent some time thinking about how do you manage data of the application, how do you manage it so that application state is kind of controlled. It's not completely different every time you run the app. If your app UI is changing a lot, obviously that also makes test automation harder and is part of an input of how much automation do you want to do? Like is maybe smoke testing to start with the right call. I think though we have seen a lot of success of people coming in and vetting it and I'm seeing it actually does work. And we try to. In all the demos that we do with customers and also with the zero step library that I think we'll talk about a little bit later, we try to give practical examples, like real examples. It's not against, you know, a test automation site that's kind of simplified. It's real things like click on the first result in Google or you know, like sign up for something like schedule something in calendly, things of that nature. Yeah.

You know, I would think as a vendor as well, it gets frustrating. How do you fight off maybe unrealistic expectations? I know as a tester a manager might hear AI tool, hey, we'll get rid of you all. We got this tool, we're all set. Even as a vendor, you might think, oh wait a minute, that's not what we're saying. So how do you fight unrealistic expectations then? If people hear AI and test automation, they think, all right, I don't need anyone anymore. They just run this tool.

Yeah, it's. I think there's a lot of. I think there's a perception that we would hear that from people vetting us a lot. Like, hey, I could get rid of testers. I don't hear that. What I hear is, I do hear kind of sense fear around what's going to happen. Is AI going to take my job? I don't think that that's the case. I think what's going to happen is that this is a tool that is going to make everybody more productive. And if you think about testing, I've been, I was a developer before and then developer manager. And then, you know, that was before we started Reflect. Testing, I feel like is unique in that the job is never done. Like there's really no one ever says I don't have enough to test. Like there's always something more to test. And so a tool that's a lever on a tester's time I think is really powerful. And it, you know, part of the fear of testers too is that, and I think this is also legitimate. Like, am I respected enough in the organization? Right? Like, you know, do developers listen to me? Do product managers understand when I say that these requirements are not met? I think that this being a lever on people's time will help them demonstrate even more how important they are to that organization. So that's my perspective on it. But obviously I'm biased because I'm a vendor, right?

But yeah, no, but I like that like you said, we're never done with testing anyway. So no matter how many tools we have, we still are going to always be testing.

It's an infinite inbox.

Yeah, exactly. That's so true. So as I mentioned, I have a new show and I covered Playwright AI for Playwright and it really, really got a lot of people talking, a lot of people excited. So the tool I source some, I think it's open source or it's free. It's 00 step. So maybe why 0 step? You have a tool reflect now 0 step. What's the difference? And why? What's. When you 0 step, what's this all about?

Yeah, so with Reflect, we're really focused on testers, mainly functional testers, but also automation engineers that are comfortable using low code and no code tools. So going back to this concept of fitting in your current workflow, that's who we're targeting with that, with folks who are developers or very have code based workflow. What we wanted to do is release a tool that allowed them to get the benefits of the AI that we've built, but fits into that workflow. And so we called it Zero Step because we don't think that there's really much overlap between the user base of Reflect and this new tool Zero Step. And we didn't want users to have basically a bad opinion of it because they see record and play and say I'll never use it. So that's why we needed something different. So it's using the same AI technology behind Reflect. It's just, it's a library within Playwright. So you can get the advantage of Playwright's Runner. You get the advantage of including playwright commands interdispersed among the AI calls. And you get the benefit of source control and, you know, versioning and all that good stuff.

All right, once again, because this is more audio, can you, like, explain a little bit more? Is it someone's in their IDE, they import the library, does it do IntelliSense for you? Does it write the code for you? Just write like, I want to log into my application, that it knows what that means in context, so it writes the code for you to do that.

So this is a little bit different than something like that, which would be more like copilot, GitHub, copilot. So the concept here is that you actually have the AI prompts inside your source code. So if you think about a playwright test, you would load up a page, a test within playwright. So you might have many tests, but a test within playwright is generally load up a page, you probably log in, and then you perform some set of actions. Those would be clicks or inputs or assertions, things of that nature. Those calls can be replaced with what we provide, which is an AI function. And that AI function just takes a string and you describe what you want it to do. So you could say in the AI function, instead of like a page click, you could say AI click on the login button or replace a series of steps that fill out a form with AI, fill out the form with realistic values. And on our website, ZeroStep.com, we have a bunch of examples of where you could use it. But why we went with that approach versus generating code is that we think that the AI is going to be most valuable in testing when the AI is running at runtime. And that's because, going back to this concept of creating tests and maintaining tests, the ideal, in my opinion, is that tests only break when there's a functional change to the app or there's a bug. And so we think that the AI gets you much closer to that than writing things that at the end of the day are based on the implementation of the page, the selectors, you know, the individual elements that you need to fill out in the form, things of that nature.

So it's quite different than, compared to, say, ChatGPT, because ChatGPT would say, or here's the locator, here's the CSS selector, or something to use. This is more, more condensed. And I seems like would save a.

Lot of code, right? Yeah. So chatgpt, I use that a lot for coding. Now you might use ChatGPT to create a playwright test, like kind of the starter test or if you don't know, I use it a lot for asking questions like how do you. What's the command to fill out form field in playwright? And it'll give me that. This is more something that's actually in your source code. And so you're. When you're running the test, it's consulting the AI at runtime to basically determine given the state of the page, which for us is what's the state of the internal page and the screenshot. How do I. What actions or assertions do I need to do to fulfill this particular prompt that's in that AI function?

Nice. So two things. Is there any overhead when you use that approach? Does it need to talk to a server with that API and then come back or is it all local somehow?

It does call a server. So it would just be like if you're using the ChatGPT API or using ChatGPT, it's going to the ChatGPT server to answer that prompt. The zero step AI is also going to our server as well. The reason it does that is because the AI model is running needs to run in our server. AI models today are generally the best ones are too big to fit on a laptop. And so just like we're using OpenAI under the covers, so we need to call OpenAI to answer some of the questions. You wouldn't be able to run that locally today. Someday in the future that might be the case. But yes, it's making server side calls.

So people that might be freaked out like I'm doing a AI.fill form, generate data for me.

Yep.

How do you trust it? Do you have good reports or outputs that say, hey look, maybe you expected this, this is what we found and this is what we did. Or how do you know, make sure the AI is honest or it's not hallucinating or just doing something when you. It's really not doing what you think it's doing.

So it is a black box we provide information about. Playwright itself has a lot of built in things to help you know what it clicked on and what it was doing. We hook into all that so you'll be able to see what it clicked on, what it filled out, you know, using the debug mode or the reports or something like that. We provide error messages when we're not able to fulfill it. But at the end of the day, the workflow within the AI tool is very similar to if you're writing selectors, you're probably going to write them, you're going to run it and you're going to watch and see what it does. And so that's the similar aspect with the AI. The thing that's different about AI is that you do, since it is a black box, you'll need to have confidence that the AI is going to be accurate enough to do those actions. And then again, that's why we provide a lot of examples to show, hey, this actually works in the real world. And all of our reflected customers or the 2/3 or more that are using our AI, it's built off the same AI, so it's production ready.

Nice. So why Playwright is it just because you have to start somewhere? I know a lot of people would love to see something like this for Selenium. You see that down the road, what's next, I guess for Zero Step or anything else, or even Reflect?

Yeah, we see a lot of applications for it. I think for us, since we're a small team, we wanted to start smaller with Zero Step, basically a large enough user base that we can see is this actually providing value and kind of nail that experience before we expand to other applications or other types of testing. But yeah, I could see it being useful in a lot of different areas. For Reflect, what we're really excited about is finally being able to fit into that manual testing workflow. And so today with the Reflect side with integrating with test case management tools, we integrate with a lot of the popular ones, so Zephyr, X Ray, testrail. But you know, we would look to expand to other test case management tools in the future. But yeah, it's just kind of you start somewhere and you expand as you go.

So how extensible is it? Say someone has some weird wonky in house test management tool. Do you have an API they can hook into and write it? Are. Do they have to work with you to get that up and running?

We would, we do have an API for the extension. We would need to add support to it. So, you know, we've been collecting feedback on folks who are using tool test case management tools that we, we don't support right now. And there's a lot of test case management tools, a lot of good options. So that's something where we're looking at. But with Reflect, even if you're using a test case management tool, if you want to use the AI piece, you can use Reflect, you can use our API to sync the data over to the TCM tool.

So going back to Zero Step, like I said, I got a lot of people really interested in it. Is this open Source. And why would you make it open source? Or is that part of it? Is it like a Cypress model where some of it's open source and the other is paid?

Yeah, the library itself is open Source, so there's two components to it. The first part is the JavaScript library that integrates with Playwright. And that's another thing that we may change in the future. We've had feature requests for adding Java support, Python support. So, yeah, so we've. We'll probably be adding support for other languages in the future. But the library, what that does is hook into Playwright. So when you make an AI call with this AI function that has a prompt in it, we send that to the. Our backend, which has the prompts and the state of your application. And then our backend is what has the AI to determine what action or assertion or set of actions to do. And so that's done through this. We do WebSocket connection, which is a persistent HTTP connection. So the library is open source. We welcome any contributors. The server, just like, you know, OpenAI is closed source.

Awesome. Last time you made a prediction, I think you said something like was multimodal AI is going to be an important next step. And apparently OpenAI's new model, GPT4 Turbo, with vision. OpenAI now has multimodule capabilities. So can you talk a little bit more by why you think that's important or how that's really going to help testing? Because before I don't think it was. It wasn't there at all. You said, I could see this happening. Now it seems like it is happening. So what could people expect from this?

Yeah, so back in July, there were a lot of rumors about what was going to come in the next iteration of GPT4, which is still the most. It's still the current version of GPT. They now have a new version of 4, which is GPT4 Turbo, and that has this Vision component. So back in July, I said we were really excited waiting for support for multimodal. Multimodal, all that means is that in ChatGPT and with the OpenAI API, the GPT API before multimodal, all you could send was text, basically. And so you could send text, you get text back. But with Multimodal now, you can send text or send images or text and images and get back images or text. So there's Dall E3, that was something that actually was released. Dall E was released prior to GPT3 and had some, you know, you saw a lot of like viral pictures about it and then other models came along like stability, AI stable diffusion, and midjourney. Dall E3 is really good and the same tech that's in Dall E3 is what's used with multimodal for GPT4. So basically what that allows you to do and what you'll see more products coming out soon about with this is being able to take visuals and take action based on that. So we're incorporating that into our model to be able to take things that normally you wouldn't be able to automate, like very visual applications or verifying images or things like that, and do that through AI, which before you really, it was really hard to do.

All right, this is crazy because I think this is the future because, you know, a lot of applications I used to work on for insurance companies and healthcare was either legacy or custom components that none of the tools could recognize. So using Image Base seems like it would, would have helped with that. But now it seems like this is even more reliable because back in the day a pixel was off, it would blow things away. So it almost seems like you won't even have to write any code with selectors per se, other than here's an image I want to get, get that image and compare it to that image and continue on and be able to take one test that runs against a web browser and maybe run it against a thick client application. I don't know. Is that.

Yeah, or the mobile app or something like that.

Oh, the mobile app, right, right, right.

Yeah. No, I think you're exactly right, Joe. It's really exciting because it gets us closer to the AI acting how a normal tester or end user would use an application. Right. And I think that's one of the things that's interesting about our prompt steps and I think would apply to anything using this vision platform is to get the AI to do what you want. You're not describing it in terms of the selectors that you want, you're describing it in terms of the end user behavior. It really comes down to those fundamentals of functional testing, which is how do I know how the application is supposed to behave and how do I test that and how do I communicate that effectively?

Oh my God. Because the resistance to automation testing back in the day is like, this is not how a real user does it. Well, I see people resisting AI and you're like, well, this is actually how a user does it. Leveraging AI, doing visual. So it almost kills a lot of those resistance, I would think. Hopefully.

I would think so, yeah. And I think it might mean unlearning some things that have been been popular in automation for a long time. I think selectors is something that, you know, talking a couple years from now, you know, new automation code is likely to not be using selectors that much. But yeah, it's going back to the roots of testing in my opinion.

Love it. Okay Todd, before we go, is there one piece of actual advice you can give to someone to help them with their AI automation testing efforts and what's the best way to find contact you or learn more about Reflect?

Yeah, so with AI and this might have been what I said last time, but just give it a try. I think it's a lot of folks. I think I would love to see disclaimers from folks who have never tried it but are offering their opinions on it. You can have your opinion, but I think you should really try it first because you'll see for yourself if it works for you and your use case. I myself have been really impressed with it. For folks who want to try out Reflect, you can visit our website Reflect Run. We're free to try. We have a free tier and an unlimited use 2 week trial for automation engineers and developers who want to try zero step. You can visit our site@zerostep.com it links to our GitHub repo and also links to how you can install it in your Playwright tests.

Thanks again for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com a4778 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space, or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.

Sam.