AI-Assisted Testing Platforms with Todd McNeal

Foreign.

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating automation awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast and today we'll be talking about with Todd, all about a assisted testing platforms, automation, codeless visual testing, a whole bunch of hot topics that I think you need to know about as we go forward this year and in the coming years. If you don't know, Todd is the co founder of Reflect, which is an AI assisted test automation platform. Prior to Reflect, Todd has worked with multiple different companies in software development and engineering management roles and a handful of Fortune 500 in a much, many different startups as well. So he has a lot of experience. So I'm really curious to dive into this topic and learn a little bit more how Reflect can help with automation as well. You don't want to miss this episode. Check it out.

This episode of the Test Guild Automation Podcast is sponsored by the Test Guild. Test Guild offers amazing partnership plans that cater to your brand awareness, lead generation and thought leadership goals. Get your products and services in front of your ideal target audience. Our satisfied clients rave about the results they've seen from partnering with us, from boosted event attendance to impressive roi. Visit our website and let's talk about how Test Guild could take your brand to the next level. Head over to testguild.info and let's talk.

Hey Todd, welcome to the Guild.

Thanks for having me Joe.

Great to have you. So the bio was a little thin, so I'm just curious to know how you got into creating a test automation platform. What led you there?

So I had worked at a previous startup with my co founder and as a software developer had done end to end testing, you know, in different roles. And at that time it was a lot of selenium. And so what we had when using those tools is just that as a developer, as a product developer, you don't really have that much time to go and maintain the tests. And so we would often we had a few cycles where we would go and try and build it and then it would work for a little while and then start failing and then we'd kind of stop running them. And so it wasn't that there wasn't value in the test, it was just that it took too much time for us to maintain. And so what we ended up doing at that particular startup, we were around 20 engineers, is we would just manually test on every deployment. And as a developer, that just felt wrong. You know, you're always looking to automate things, try to remove the manual processes. So that was really the impetus for us to start the company.

Nice. So I know there are a lot of automation tools out there beyond Selenium. What made you think for what we need and what we see in the market, needing what makes reflect different, then why did you create it to be maybe address issues that other tools out there weren't addressing?

Yeah, there's a lot of tools in the market and I think it makes it for testers and folks evaluating the tools, it's difficult to really know one from the other. A lot of tools make the same claims, but I think that's because everybody faces kind of the same issues when you're doing automation. It's the speed of creating the test and then the ability to maintain it over time. I think a lot of engineering organizations, they're trying to move faster and nowadays you're trying to do more with less. There's just less time, there's maybe less people than there were on the team before, and it's hard to get everything done. And so what's different about us is that we really focused on the two core problems, making tests fast to create and easy to maintain. And so one of the things about our tool is that when you're recording record and playback tool, but when you're recording, you're doing it in a cloud browser, you're not doing it in extension. We do that so that we can control the infrastructure and handle things for you that maybe other tools couldn't. So things that would be a best practice if you were a Selenium developer or a playwright developer, doing things like auto retries and, you know, detecting network requests before you proceed to the next action, you might need to hand code some of that or know how to tweak things. We just do that on your behalf. And so it's a lot about doing things on the user's behalf that they would have to normally write in code.

Dive in a little bit more about that. So, like, what feature would that be? So someone records and a lot of times they have to actually add a wait.

Yeah.

But because they're running in your platform or your SaaS Cloud, it knows that the application is completely loaded without the user having to know or explicitly call it out. Is that how it works?

That's right, yeah. So there's a couple key use cases for It. So the first is on page load. And this is a lot of platforms do this. They'll wait for the page to be ready before you interact with it. Kind of the next level is, let's say you do some interaction like a click. How do you know when it's ready to do the next action? A lot of applications right now there isn't a next page load. It's all single page apps and very dynamic. So what we to do, which I'm not sure if any other tools do, is automatically detect things like animations, a network request, or set of network requests that happen after that click. So if you do a click, it sends a network request and then two seconds later that network request is finished. We'll wait for that to complete before continuing.

Cool. And it does all under the covers without the user having to configure it themselves. Very nice.

That's right.

Cool. So what I think is also kind of interesting is you're a developer. You start off as a developer, and the solution seems to be like a codeless solution. I know. Codeless has become more and more popular. Yep. It almost seems like someone that's an experienced tester goes, oh, I'm a developer, I don't need codeless. But you are a developer.

Right.

And you create a codeless solution and you would have used this as a developer. So why codeless then?

I think that's ultimately the best way to solve the problem. We didn't start saying we want to build a codeless tool. We started and said basically, if we could build a solution, that would be the best tool for solving this problem, which is the test breaking. And it's hard to keep up with the pace of development. Really what you need is something different than the code base paradigm that's been around since Selenium. You can make that work if you have a team dedicated to that test. But for a lot of organizations, they don't have that and maybe they don't want to do that. And so you need something at a higher level of abstraction that will actually assist you in managing the things that you know. Otherwise you'd have to go in and do yourself.

Absolutely. So besides codeless, I believe you also do visual validation testing or visual testing.

Yep.

That's something sometimes not everyone has. So why add that feature?

Yeah, so we added that really early on and that's been a really key feature for us. Again, it just kind of came down to like, if we look at, we started with our test cases that we would have in our company, and then we looked at as our customers came to us and said, here are the test cases we need to support. A lot of times when someone's evaluating a codeless or any tool, any automation tool, one of the key questions is can it actually automate our existing test cases? Right. You want to know that you can go from 20% coverage to 80% or more coverage and not hit some limitation in the tool. And a lot of times what that means is that you need visual testing. Because if you would look at a manual test script, it would say, you know, click on this, validate that on the next screen there's, you know, a blue button, validate that the thing appears on the right side instead of the left side. You need that level of expressibility in order to really meet your requirements. So yeah, it was there from very early on.

Nice. So I know sometimes people shy away from image type of validations because sometimes they become flaky as well because of little pixel differences. Because it's running in your platform, does that make the approach you're doing better or I'm just guessing, I don't know.

Yeah. So by running in our own infrastructure, we have a lot of control over how we do the visual validations. So one area where it can get really flaky is if you're doing visual validations across cross browser because each browser renders things differently. And so what we do in our platform is we run the infrastructure to run the tests. So we're running the Safari VMs and Chrome and Firefox, et cetera, VMs. We create individual screenshots for each browser. And so you can think of the first run of say, Edge as capturing the screenshots for Edge. And so that helps it so that if you're not cross comparing or using something you captured in Chrome, it helps reduce the flakiness there. But there is, yeah, visual tests. If you're not careful, if you use too many of them, it will be flaky because you'll basically get notified of changes that you don't care about. So what we advise is when you're thinking about doing just a regular text validation versus a visual validation, think about like, if this changed, would you open a bug ticket and like, what severity? Would that be trivial or would it be like, you know, major? And that should kind of guide you as to whether you want to do visual or not.

Gotcha. So I know a lot of testers I speak with, not only do they use tooling, but they also use like browser platforms in the cloud to run against all the different devices?

Yeah.

Do you do the same or as this just happens to run in the browser but you don't necessarily include all the infrastructure to scale. How does that work?

It's all included in our platform, so you don't need to use an external test grid at all. It runs. We handle that all for you. And one of the nice things about that is too is that the same infrastructure that we use when we're spinning up to record the tests because it's in the cloud is the same infrastructure we use to run it. So you have less of these issues. Where I recorded it locally in an extension say, and then I try to run it in CI and CI the environment's there's all sorts of differences from maybe it's underpowered to the browser dimension is different or the browser version is different or you're in a VPN that this approach factors all that out. So it removes some of the friction of getting tests running locally to running in CI. Perfect. Yeah.

So I also know a lot of times people just focus on UI automation. What are your thoughts on API testing? Does your platform handle it? Like could you have any best practices of when to use API testing versus like a visual type of UI test?

We do support API testing in our platform and one thing that's unique about it is that you can have API calls side by side with a UI call, clicks and inputs in the same test. What's been interesting about that is the use cases that have emerged from our customers doing that. What it's allowed some of our customers to do is be able to really reduce the size of a UI test. Because whereas before they might be using the UI to. Let's say the UI test is I want to edit an existing record, so I need to go into my test application and edit it and that's a destructive action. Once I edit it, it's changed. Maybe I need to go back and reset it or something. With an API call I can go in and create a new record with a single API call and then go and edit that record directly. And because my test is creating new resources every time, it doesn't matter if it fails in between, it's not going to affect the next test, which has been really, really interesting. The other use case that we see is a lot of applications are integrating with third party things, whether you're an E commerce site integrating with stripe or shipping or tax integrations or a SaaS platform, again with payment or other things you don't want to be have to go to a third Party application in the UI to validate something. In those cases you really need to just make an API call to validate that the order is in stripe or the shipping rate matches what is in I backend system. So API calls are really helpful for that. Great.

So you know, besides API testing, you know, we talked a little bit about codeless. So a lot of times testers or developers like to actually see the output and be able to change things. Like how customizable is this? Are people forced to use codeless? Are they able to see the code that's generated in the background? They need to customize it. How's all that done?

It's completely abstracted away, except that you can have code steps in your tests. So it's not one way to think about it is it's not generating like Selenium code and running that code like something like Selenium IDD or other tools like that. It's generating something that can run within our Reflect runner, but you can add code to it if you need to express something in code. So you could do use code to extract cookies to validate things like that. Or maybe there's something that we don't support. You could do that in code.

So would it be like a JavaScript code step or something they would drag on?

Exactly, yeah. It's a JavaScript code step and it supports synchronous or asynchronous calls. So you can even do things like network requests or use any JavaScript API that would be asynchronous, which a lot of the new ones now are by default asynchronous.

Yeah, so a lot of times people struggle with speed of tests because it's running the way in your special infrastructure, do the tests run faster? Is that possible?

Generally they do run faster, yeah. When we've benchmarked it against other tools. And part of that is because, yeah, we control the infrastructure so we can make investments in things like faster startup times. We can do things that would be not possible to do if we relied on an external grid. So for example, we have faster startup times because when we load up our infrastructure, we pre cache the browser sessions. So every browser session is isolated. You'll never have, you know, we destroy the session after it's complete, but you'll have that browser session already running and so it'll just be a second or two in order to jump into the session. Another cool thing that we can do with the infrastructure is jump into a running test. So you can actually run a test and click a button to Watch it live and actually interact with that test live, if you wanted to, to, like, fix something up.

Very cool. So it's like a live breakpoint, almost sounds like.

Yeah. And in fact, that's what we call it. You can add breakpoints to it, you can pause it and kind of move where you are in the test. So if you wanted to add steps near the beginning, but you've ran to the end, you can just kind of drag this cursor up and say, now I'm at step five. Yeah, cool.

I like it. So I guess another thing that's big, obviously, is AI. And it sounds like you started off, you've been around before incorporating AI features, but I think I noticed you created some AI features.

That's right.

So I thought maybe we dive into. AI is obviously a huge topic. Some of the AI features, I think you have something called like an AI prompt, steps and AI assistant, I believe. So can we talk a little bit about those AI features, why you created them and what they do?

Yeah. So we hadn't incorporated any AI features up until a few months ago, just because we were focused on the reliability of the test, the accuracy of recording, and just kind of getting the breadth of features so that customers can meet their requirements. But once we started playing around with ChatGPT, we realized this is a pretty big game changer, I think, not just for testing, but for a lot of things, but certainly for testing. And so our AI features integrate with OpenAI using the same technology that ChatGPT uses. And the first feature, like you mentioned, is AI prompts. And it's basically, you can think of it as an arbitrary action or set of actions that you tell reflect to do. So it doesn't require any sort of syntax. It's not like conforming to Gherkin or any sort of custom syntax. You just write what you want it to do, like click on this button or, you know, input something and it does it. And what's cool about it is that you can use anything that OpenAI's model is trained on as part of your input. So, for example, if you were in healthcare application, maybe there's some things that are specific to the domain of healthcare that you need to enter. ChatGPT is trained on lots of data sets, including healthcare data sets. So you could enter in things that maybe only someone with specific domain knowledge would know. Like in some of my demos, I show, like, you could enter in the median salary of nurses in the United States. You could enter in phone numbers, sample phone numbers from any country, and it Knows the correct area code and kind of length of the phone number to use. So there's a lot of things like that you can do that is cool.

So you can almost say like given this patient, give me the top values for blood pressure or something automatically.

Exactly.

That is wild.

Yeah. And so what that allows you to do is a lot of the things that are already in your manual test scripts or in your requirements, like the specs that you have, maybe you're doing bdd, you can just take that and put that into a prompt step. We think that eventually you could take manual test scripts and you know, get 90% of the way to an automated test without any sort of user interaction. And so that's kind of the direction that we're heading right now.

All right, so when I saw it, it's just a text box and it seems like pretty free flowing. Like how, I mean I can get pretty crazy with my chatgpt prompts. Like it's just like that, that you could just type in whatever you're thinking and it will automatically. I mean obviously it'll try to give you what you want, but it's that free flowing.

Yes, it is arbitrary information. And basically what you can think of it as, it's a freeform prompt that results in one or more sets of actions in the browser itself. And so today what those resolve to are things like clicks and inputs and things of that nature. What we're adding to it is the ability to do things like assertions. So if you would again think of a manual test script, you would have things, you would have assertions intermixed with these inputs, like go to login and then verify that my name is on the top. Right. That kind of verification is going to be something that you can enter in a prompt pretty shortly in reflect. And then another thing that we do is we'll use the AI to determine if what you entered is actually a set of steps and then we'll split it and show it to you. So you could actually. One thing that we've seen our customers do is basically copy paste a whole list of steps and then it will show you the individual steps and allow you to edit them if you want and then run them right after.

Wow, I'm still really digging that test feature data thing where you can just have it just create realistic data for you based on what it knows about that particular segment. So that's really cool.

Yeah.

Is that used a lot? I mean, your customers, is that something you've seen as one of the killer features or the.

Yeah, I would say so far it's been pretty early, but what we've seen from our customers so far is that's definitely one of the features they like. Another thing is the expressibility of it. So there's some scenarios that just if you're not coding, it's just very hard to express. And I think automation folks have had legitimate criticisms for no code low code tools about this. Like an example would be I have a table of values and I need to click on something associated with a row and it could be anywhere in the table. That's just very hard to do without being able to drop down the code to do it. But with AI, it turns out you can do that just by expressing it in English language. So you could say find the row associated with Joe and click on the edit button and it will work whether you're on the first row or the tenth row or the seventh row.

Wow. Yeah. Absolute tables tend to cause issues. And also graphs. I don't know if it can help at all with graphs or anything like that.

Yeah, right now it's not very good at graphs because it doesn't see the canvas. What's missing right now with the state of the art and AI is what's. There's no off the shelf AI. That is what's called multimodal, which means it can process text input as well as image input once that technology is released. And so OpenAI said maybe early next year some other ones that are competing with it are trying to get there earlier. That will open up even more use cases like graphs, interacting with graphs or some things that are very visual. So like if you were a gaming application. Right. Where everything is on a canvas, like a slot app or something like that, very, very hard to automate that right now. But with a multimodal AI, that will now become possible because it will see the visuals as well as kind of the text information.

Nice. It's the first time I heard of multimodal. I'm going to check that out, but that seems like once that's live, that definitely is going to be another game changer.

Yep.

So besides AI prompt steps, which is to me sounds really, really cool, people need to try it out for sure. Actually for people that want people to actually be able to see this or try it. Do you have a free trial at Reflect that they can do it?

We do, yeah. So our website is Reflect Run and just type that in the browser and you can sign up for free. Our AI features are launched to everybody. So even in the Free version of Reflect, you'll be able to try it out. And when you sign up, you get a free unlimited trial for two weeks. So there's no restrictions on how many tests you can run for the first two weeks.

Cool, Cool. All right, so besides trying that out once they have that free trial, I think once again you have an AI assistant. I really like this approach. A lot of people are coming up with using AI to assist, not replace type deal. So yeah, in the name itself, I guess it tells what it is, but could you just explain a little bit about what the AI assistant does and reflects technology?

Yeah. So our AI assistant, you can think of this as like a next generation self healing. So self healing technologies have been around for a while and they're basically to work around problems with selectors, where a selector, like an XPath or CSS selector, uses the page styling, the page structure, to determine where to interact with something. And one of the main frustrations with automation is that as the page changes, those selectors can break and you won't know until your automation runs. What the AI assistant does is it uses the same technologies or AI prompts to find the element when all of the selectors are invalid. So in reflect, when we record tests, we generate a English language description of each test step. So like click on the submit button, input Joe in the first name field. And with the AI system actually takes that and uses that as the prompt. So by using the same approach, basically the better your tests are documented because you can edit these descriptions, the more information the AI knows in how to perform that action when the selectors are invalid.

Cool. So I guess the output then is very readable as well then because it will all be English like and the up, you can just hand it to someone maybe that's new on the team and say here, here's what the test is doing.

That's right, yeah. And we've heard from several customers that they use Reflect for onboarding, new folks, even people outside of the testing organization because it's very. You have a video, the video is synced up with the steps that we generate. And so you can see, you could go and see like what does registration look like for this application? You can actually very quickly find it and then watch a video of it. And one thing that we do with our AI as well is that in some cases it's hard for us to generate an English language test description ourselves. So if you click on something which doesn't have any text in it, like a logo or something in a graph or something like that there's no text information for us to go on. But what we do is we ask the AI to give us a description. So when you click on it, we tell the AI, here's the state of the page, here's the action that the user took. How would you describe that? And so you'll see it say it clicked on the logo in the top left, it clicked on the highlighted region in the map. And so you have that auto generated description for later when you're looking at the test and kind of forget what it's doing or when the AI needs to go and figure out where this thing moves on the page.

So does this help with multi languages? I know a lot of times when I had, we used to have an application it was in English, but then we'd had a test in all the different regions that we sold the application to and test all the different languages, outsource it, get the translation and then check if the translation was correct. This seems like a good use of AI. I don't know if this is something your application does is only handle English.

The AI is able to understand different languages. So it's internationalization is a good use case for it. Also a common one with internationalization is wanting to run the same test case across multiple languages to test for your translations. It's not an AI feature, but within Reflect you can parameterize all of your tests so that you could basically create a spreadsheet and say for this particular test, I'm going to run it 10 times for the 10 different languages and then validate that all the translations are what I expect.

Nice.

Yeah.

So are there any current limitations with the AI features? I know a lot of times tools have issues with iframes or shadow doms and things like that.

Yeah. So right now with our AI feature set, the big limitation is the lack of assertions and that's something that we're adding in our tool in the next couple of weeks. We do have support for Shadow DOM with Salesforce, so you could go in and use this for Salesforce and that's a great use case because Salesforce has a lot of these forms and with a lot of fields and tables and grids and things of that nature. IFrame Support, we're adding in the next couple weeks as well. So I would say in the next month there shouldn't really be any limitations in terms of what people want to do. Probably up until we add, probably the last limitation is going to be that multimodal approach. Yeah. So until we see the Computer vision. There's going to be some use cases we can't handle.

But yeah, so as we mentioned, there's a lot of tools out there. I believe different tools fit different teams better. Not one's better than the other. Just certain situations lend themselves to different technologies. So.

Right.

What would you say reflects target user is or you think someone listens to this that falls into this category would really should jump on this right now.

Yeah. So one of the things that was challenging, I mentioned that I came from a development background and so we built this for us first. But one of the things about building a business is understanding who your user is and you really want to find the users that love the products. One thing that we found is that Automation engineers and SDETs folks that are doing coding, they just want to code. I think that's kind of the reality is that it was kind of a hard one. Lesson is like it doesn't really matter how good Reflect is. A lot of SDETs and automation engineers just want to code. And so we see our user as primarily the tester who is not doing automation today or you know, doesn't want to code but wants to automate. Those are the two users that tend to love this tool. And increasingly we're seeing front end and product developers like full stack developers using it because it's a time saver so they can own end to end testing without actually having it take up too much of their time. Great.

And Todd, this is mainly audio only, but I know we're actually doing a webinar on July 25 that's going to go over building automated tests using generative AI so people can actually see this in action. Maybe a little teaser for people that are going to join us on that webinar what they're going to actually see. So they can actually see if this is really going to do what it sounds like it's going to do.

Yeah. So I know a lot of folks are exploring ChatGPT and other AI technologies and you may be exploring technologies that allow you to do work faster. So maybe generate test cases or generate Selenium scripts or Cypress scripts faster. What we're really focused on is kind of the step change in AI, which is what would it look like if the AI can be your assistant in building out tests. And so what that really means is being able to record a test and have it work even if there's large scale changes in your application. It means pulling in your manual test scripts and getting 90% of the way to getting a fully working automated test and it's allowing you to express things that right now no code and low code tools have a very hard time expressing, which means that your percentage of things that you can automate with these tools becomes even higher. And so that's really what we're focusing on is showing that in Reflect and these are all features that you can sign up for and use for free today.

Yeah, I highly recommend anyone listening to this check out the webinar we'll be doing on July 25th. Even if this is after the fact, it'll be available for replay and you'll be able to find that at testskill.com forward/webinar and hopefully we'll see you on the actual event because I think it's going to be a lot of value and I think we're giving away a free book. So if you do attend, you will get a free book. So definitely something you should definitely check out. Okay Todd, before we go, is there one piece of actual advice you can give to someone to help them with their automation AI testing efforts and what's the best way to find contact to or learn more about Reflect?

So I would say one piece of advice I would give for folks thinking about automation is thinking about how to manage state within your automation tests. One area where things get flaky is just kind of the tool and handling things that change naturally about the application. But the other area where test flakiness comes in is the state of the application changing over time. And so if you can manage that, that makes your test much more resilient to changes. And so within Reflect that would look like things like using our API testing feature to create data as part of the test. It would be making sure that you build your test so that they don't rely on another test running first and then this one running if at all possible. But yeah, that's what I would recommend and I'm happy to share any additional information you can find me on our website reflect RU N. My email is toddelect run and I'm on LinkedIn and a few other places.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode hand on over to testguild.com a45,6 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguilds.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.