SmartDriver Let AI Do the Heavy Lifting with Chris Navrides

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Chris all about Smart Driver, the state of AI and testing and what testing may look like in five to 10 years. You don't want to miss this episode. Chris has a lot of experience in this area. He was the VP of engineering at Test AI, which built the first AI based mobile and web testing solution. And before that, Chris actually worked at Google on automated testing for Google Play and for mobile ads and Dropbox on their mobile platform team. So Chris is an expert in this area. Really cool solution that's coming out that's been, I guess he just released it. Smart Driver that's currently free to use and you definitely should check it out after you listen to this episode. You don't want to miss it. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit testskill.comSourceLabs and and click on the exclusive sponsors section to try it for free today. Check it out.

Hey Chris, welcome to the Guild.

Oh, thanks. Thanks for having me. This is awesome.

Cool. Great to have you. So Chris, before I get into it, is anything I missed in your bio that you want the Guild to know more about?

No, I think you covered everything.

Awesome. So how did you get into AI?

So funny story, I was actually working at Dropbox at the time and I was building some bots to do testing. And so we were building bots and sort of the tree of the entire application there and then, you know, having bots go through and test the whole app. And so I went to a conference and met Jason Arbin who was the founder of Test AI. And so we hit it off and you know, over a couple of drinks at the bar late one night, you know, where he was talking about AI and so that was sort of my intro to it. Eventually he recruited me out. I was at a back at Google at the time and so he's like, hey, why don't you come, you know, work on this thing? We're going to do AI for testing and actually solve this problem. And so got me over there and so I like to say I was AI, formerly AI curious, and then dove in with both feet and kind of looked at the industry, that area and went from there.

Nice. So a lot of people's dream job is working for a company like Google. So what did he say that made you go, okay, I'll do it? I know Jason's very persuasive. Was there anything specific that he said?

Just blinking those eyes?

No, it's baby blues. I don't even know if he has blue eyes.

No, I think it's, you know, Jason is just a very passionate person and I think that's what him and I resonated on is, you know, him and I both had this shared experience that we want to just solve this problem. And this problem is that testing in general, that testing frameworks haven't really advanced in about 20 years. You're still for UI based testing, you're still going in, you're building selectors for elements that are bound to change. So you're going to be doing this constant maintenance and you know, AI is finally advanced to that state where, hey, things can finally get, you know, get good enough that it's starting to surpass what I think traditional automation can do. Just looking at the UI and the visual side. And so he was kind of talking about this sort of vision and getting me excited and you know, we were kind of talking about why isn't company like Google or Facebook, some of these big mega corporations doing it. And I think what we kind of discovered was look at most companies, testing is an afterthought. If you look at testing across the board, it's sort of thought of as, you know, the reason why you can't launch on time. And so people don't tend to want to invest money if, if they have, you know, $10, they'd rather put it into the new feature which will help drive revenue as opposed to necessarily, you know, hey, building a better test framework or putting an extra higher on the QA team. And so that's where I think the startups and especially some of the new companies coming out are, you know, fully dedicated to that one mission. And so that was what really got me of like, hey, we had a chance to couple shots on goal to try and solve this.

Nice. Now I was going to save this question till later, but I know your company recently got accepted into Y Combinator and you're like the second testing company to, to be that I've spoken to recently to get accepted. When I think of Y Combinator, I think of, you know, not real startups, but you know, like startups and like social media kind of companies, you know, but it looks like they're starting to take testing seriously. And you said, you know, companies don't seem to be taking testing seriously. So have you seen a change in the way maybe angel investors and investors are looking at this as a real issue as well?

Yeah, I mean, talking to investors, I think the number one question that kind of keeps coming up is why is there not another Mercury out there? And so Mercury was, you know, about 15 plus years ago sold to HP. It was a $2.5 billion exit. I think if I recall correctly, since then, the amount of money people spend on testing, testing services, you name it, has just grown so much. And yet the whole field is very fragmented. There's not kind of one major company that's, you know, owning it in the same way that you see like in the transportation space, there's Uber and Lyft. Right. And they're massive. If you look at search, there's Google and bing in the U.S. those are massive. You don't tend to see that. Instead what you see is just a whole bunch of different companies from each different sort of vertical. And they don't tend to massively go outside of their vertical. Some of the private equity firms are starting to sort of coalesce a lot of these testing services into one company. And so you're starting to see some of those. But there isn't a clear winner. Like there is, you know, with an Uber or a Google or something like that. So I think that's what YC is kind of looking for. And they're looking for smart people to solve problems that haven't been solved. And they've, they've found a lot of companies or I should say invested in a lot of companies in this area. And so we're lucky enough to be one of them.

Nice. So what's it like pitching in front of there? What was your, your mean? I guess you have a certain amount of minutes, not have a long time. You need to really hit it. So what was your, your killer pitch that got you in?

It was actually just a very fast 10 minute interview. It's, I've never been grilled that hard. You know, they just pepper you with questions about why you, why now, why this market? How are you going to be different than the last companies? How, you know, all of those kind of questions to just really see, like, hey, are you going to, you know, persevere? Because I'm sure you've seen this, Joe. You, you started your business and you, you get told no a whole bunch. It's, you know, Hard to get that initial traction. They want to find folks that are willing to commit, willing to be mission driven. And so this is something me and my co founder at Teen are very passionate about. I worked with him at Test AI as well and we're both very passionate about this space. And that I think came through and I think talking to and talking with some of the other founders, especially in the testing space, you tend to see that as well.

Now there's a lot of tools. I get hit up at least twice a week by a new vendor that's a codeless AI based solution. You know, first of all, you've worked for an AI company. Is this legit? Like when people say they have AI, all companies using real AI and if they did, why, why wouldn't they get all the market share if it really was AI machine learning? I don't know if that makes sense.

But like I think so. Let me take a shot at answering it. I think the way I view testing is that all testing, regardless, if you're talking API, manual, UI testing kind of breaks down into four components, right? So you have selection. So unlike ui, that's element selection, building a selector, you have an action, so a click, an API call, whatever, you have a verification. So these are your certs. This is, you know, checking whatever you want to make sure that happened, happened or didn't happen. And then you have sequencing. So what is the order of these? And so a lot of these companies, depending on where you look at them, they apply AI to various areas. So if you start to kind of look at these vendors, I'll just kind of throw out a few. Right. Like applitools is just AI verification at the UI layer. And so there's all these different kind of permutations then that go in there. Test AI, try to do all sorts of these pieces with AI and various vendors do it. I don't, I'm not going to claim to be an expert on all the different products out there, but they all try and apply AI in these different four areas, in my opinion, to varying degrees of success. Just because there's, there's, you know, solving the problem and then there's productizing it and making it, you know, useful and accessible to everyone. And those tend to be two separate challenges and require sort of two different thought processes around it.

Nice. So you did start a company called SmartDraw. Is that the name of the company? No, it's DevTools. AI.

Yeah.

And the solution? Smart Driver.

Correct.

So what's the deal with Smart Driver? What's that all about?

So similar to, like I was saying, there's, you know, those four different areas, we're really just looking to apply AI to just one area, which is element selection for UI based testing. We kind of think, hey, AI tooling right now, especially in the visual field, the recognition aspect of that is actually probably the most furthest ahead in terms of AI field. There's a lot of research, there's a lot of open source work around it. And so us being able to apply that in the domain of just say web testing to find, you know, your element that you want trained specifically for your application, we think it's, we can do that in a very smart way where we basically say, hey, if your selector fails, we can go ahead, take a screenshot, look at it like a human would apply visually and actually locate the element and then return it so your test can continue to pass. And the whole rationale for that is, hey, if it looks the same to an end user and you're doing a UI test, it should probably pass your underlying object. Shouldn't be the blockade for your test to pass if it looks the same to a user.

Nice. So does it learn over time? Because I'm just thinking back in the day we used, we used to say, all right, if you can't find the identifier, it's to the right of this image or to the left of this. How different is that from that type of logic?

So it's similar, you know, I think it definitely takes training data within it. So the more training data the better, especially if it's very good training data. And by good, I mean, you know, realistic. So if that button tends to move, it'll start to learn those properties of where it is. There's certain aspects that I still has a hard time with though, to be very frank, you know, so I like to call it the sort of Excel problem. So if you're trying to say, like, I want to select this item on the grid and it's all blank cells except for, you know, the row headers and the column headers, then how does it find it? And so all those cells look the same visually. And so, you know, this is kind of where we're trying to go. I mean, it's not solved right now, and I haven't seen it solved anywhere. But this is a common issue where it's like, you have to kind of figure out what's interesting about this particular cell is it's, you know, say in the row, first name and the column of, you know, Last name or something like that, or what have you. And so that is how you have to figure out that that cell is the interesting one that the user wants. And so sometimes that takes a lot of data. But I think, in general, the way I think of AI testing versus, you know, traditional testing is when you write your test case with the traditional test framework. So Selenium, Appium, Cypress, what have you. The best your test case, in my opinion, will ever be is the day you check it in that hour, right? Because you have locked that state, you've pulled the main branch, you're fully there, you know all the variables and it runs. And then it can only get worse from there as people, you know, add more to your tests, add more to your selectors, add more to the product, you know, add more feature flags, et cetera, it only gets worse. AI kind of flips that on its head. AI tends to get better as it progresses, so you may have to add a few samples to get it to learn what this object is or what these elements are on the screen. But after it learns that, guess what, if it, if there's a small variation, right, it moves to the right of this object. It can still understand that you're looking for, say, the shopping cart button or the search icon button, what have you. Because it's seen enough of those examples throughout your application that it knows generally what that is, and that tends to be closer to what humans do. Anyways, if you hired a new tester for your team today, they're not going to be that great to start with, but you give them a couple examples, they learn, they get better, and then they become a really strong tester. That's sort of the promise of AI.

Nice. So how does someone get to use Smart Driver? Can you give an example of how to use that? Like, say, an existing framework?

Yeah. So right now we support Selenium, Appium, we'll have Cypress out here real soon and then we're going to continue. But essentially you just install it with whatever framework package you're using today. So npm, pip, maven, and then you import our package and then you wrap your existing driver, and then what we do from there is we basically wrap all of your selector methods in a try catch with that, then we go ahead, and as soon as it fails, we basically kick in, we take a screenshot and we look for that element visually, we do that on our back end and then we return it, and then we return your element and it works just the same. So it's the same element. It Finds it on the screen, locates it. And so if, if your tests, you build really great selectors. Awesome. We have no real perfect to you. We're not kicking in, we're not doing anything really. It only kicks in when there is an issue. And so any perfect you would see, you'd already have a perfect because you couldn't find that element on the screen. And so we're trying to be just purely additive and purely just a backup helpful solution to folks.

How much time does it add if it does fail and it goes through this?

This process takes about two and a half seconds. So most frameworks, the. A large bottleneck is just getting the screenshot itself. So that's about a second. Then it has to upload it, run the inference on the machine learning model and that takes, you know, another second or so and then just kind of all the underlying stuff. So finding them, the element on the page, given the coordinates and things like that. Cool.

If someone's an automation engine, they know why this is cool. But you know, what is the main benefit you get from using Smart Driver?

It's actually a really good question. So I think there's a couple of benefits. So I think the first one is that you're not going to have to spend as much time maintaining tests. So as tests, you know, grow and mature and sort of your framework, things change. So that could be maybe 80 flights going on with certain elements within your test, you know, and so you would have to go in, put, you know, an if else type of check if this element exists, else if check, you know, if this element exists and you know, for every single sort of permutation. What's great about Smart Driver is it'll actually learn from that automatically. So you don't have to change your test code, you don't have to do another print, it'll just learn, oh, hey, the, you know, this button can be red or blue depending on which experiment it's grouped into. You don't have to modify your code if it still looks the same. Say the experiment is just maybe you're changing frameworks and so the page is being rendered with ASP Net and you're switching to react. And so the page object models are changing, but it looks the same to your user. You don't have to change your test framework. So that's the big issue or the big time savings that we have. And we've talked to a lot of companies, you know, as they've grown their test framework over the years, you know, their team has grown and their team is now, you know, spending more and more time just maintaining tests. And so we'd love to have those folks be able to, you know, not maintain tests as much and be able to kind of use this as an assistant to kind of, hey, let's keep the test green and then let's give some data about. Hey, these tests are flaky. Here's some selector options that maybe could work a little bit better. So you don't even have to use our service, things like that. We have a secondary benefit as well that we're trying out which is if you want to just skip building selectors at all, you can just use the AI. So you just. We add an extra driver function to the methods of find by AI and you just pass in sort of a human readable string. You go into a web UI and you can basically teach the bot, hey, this is what a search icon is or a menu icon or what have you. And then you don't even have to build a selector to begin with.

Oh, that's cool. Is that newer and has that been tried and true proven?

I mean it's the same logic. It's just. Do you. It's the same thing that happens in the catch exception when your thing fails. So if you want to just not have to build a selector to begin with and just use the AI, you can use that. But the method itself has been proven. We're using a similar technique at test AI and this is also similar to a lot of open source work that we're kind of standing on the shoulders of giants and that's most of AI out there is doing this where you can kind of leverage what folks have done in the field at, you know, companies like Google and Facebook and they open source it of like finding a dog on the screen and you know, large pictures, kind of, kind of similar approach to then finding your submit button or your login field. They tend to have very similar overlaps. And so that's what we're able to leverage is, you know, all the research and all the really smart folks at these large companies.

So you come out of test AI. Test AI was doing a lot of things. So when you start a new company, how'd you know that this is the sliver that you're going to start with? Is this something that's manageable or is this something that you see was the biggest pain point? Most like 80% of people using maybe test AI were struggling with.

I think, you know, after test AI, test AI had a lot of really cool technology, a lot of great people and lot of great customers. And what we saw there was, you know, a lot of folks were hesitant to jump into a full fledged framework that would have this, you know, kind of problem with say, rip and replace or you know, you have to start over on your test automation. And so with that, I think it's a little bit harder to get folks to try it out and use it because they have to redo all the work that they've spent, you know, years, months, weeks building. And so what's different and what we thought afterwards is, hey, what if we can just go ahead and meet folks where they are today and say, hey, let's just try and help you be more efficient, not have to spend as much time maintaining your tests and what would that look like? And if we could apply some of this technology and these thoughts, take the latest and greatest that's out there, do some research on what's been changed in the last couple months, even in this field, and then apply that. And so our thought was, hey, let's, let's go ahead and just integrate with these frameworks. So then guess what, you only need to change, you know, two lines. You have to import our stuff and wrap your driver and you get these benefits of AI without having to, you know, do all this hard work. You have to rewrite your tests, et cetera. We just go ahead and can find a way to learn from your existing tests and then build backup selectors for those.

It's almost like a bog approach. You're gonna simulate one function at a time, so eventually you're like, ah, I'm a bog.

Exactly, exactly. I mean, I think that the key thing that we thought just chatting with some folks early on is, has been, yeah, it's if we can just learn from your existing tests what you're using, then guess what, you can automatically jump ahead as opposed to having to have someone go in and reteach the bot. And so that's where we've tried to keep it as simple as possible and just hear value added.

All great. So just curious to know, with your experience in AI, you're speaking to all the hipsters now, probably in Silicon Valley. What's the future you think of AI? Some people have opinions like, oh, it's just a, it's not truly AI anyway and you know, it's never going to be able to do anything that, that great. But do you see it more as like a working with people to bubble up insights or is it more like it's eventually going to take over more and More and more of what an automation engine is probably already doing.

I kind of, I think that the best place that it would be is, you know, it's going to be a tool that automation engineers can use to kind of be a force multiplier for themselves. And so it's not necessarily going to replace testers or automation engineers, but it's going to allow them to, you know, focus on the hard problems and really allow them to scale themselves and, you know, have more impact across the company and organization. And I think, you know, the kind of key things to think about here is, you know, lots of times when people think AI, they think like, oh, you know, like when it reminds me, I guess, of the same thing when ATM machines came out and everyone's like, oh, the tellers are going to, you know, lose their jobs and. But when you look at it, historically speaking, there are now more tellers than there have been ever, and ATM machines are prevalent. And so it's really a win, win type of scenario. The other kind of thing to think about too in this is AI is fundamentally a data game. And so you see AI being applied to the product side. What you see is like kind of new positions where it's like, oh, these data engineers come in and they're basically, you know, Netflix is a great example of this. But you have these data engineers who, who are looking at all the data and helping train the AI versus content selection. It's not replacing the software engineers who are building, you know, all the infrastructure on top of that to make Netflix great and stream and all those aspects. It's just sort of an augmentation. So it helps, you know, the end user, the data engineers tend to be still software engineers who are writing a lot of code. And I think the same thing will be, you know, happen with automation and test engineers is there's going to be these folks who look at data and these data signals could be product insights. Right. So what are the paths that users are going through your apps? Right. And then instead of having like metrics around code coverage, it would be metrics around what percentage of our user flows today are covered with automation or covered by this build. So if we can say, you know, 95% of the user flows within our application are now covered with automated tests. That's a very, you know, very true and interesting signal to upper management and the team and that like, oh, cool, there's only maybe 5% of folks who might see some issue and hey, how can we increase that or how do we, you know, augment that Automation with manual testing or some other sort of testing. And you can start to use that. You can start to leverage data around, hey, what are core outages that we've seen in the logs and can we replicate those and try and, you know, permeate them? Like maybe service A failed, but service B didn't. What happens if A and B fail at the same time? Does that cause a catastrophic error? You know, you can start to leverage some of these data aspects as you test. And that's, I think, sort of the future where AI and machine learning can help do that and help scale that. But then it allows that automation engineer to focus on, say that, you know, 5%, that 10%, the new feature that's coming out. Instead of, you know, maintaining and making sure that we didn't break anything with this release, they can say this new feature is really ready for end users and customers to use.

Cool. Is anything AI can maybe find that a tester would never be able to find? I've been reading a book. They used AI to discover a new vaccine seen they just did parameters and it just ran through like a bunch of things and they found things that a human wouldn't even make the connection. It was able to find connections that weren't even thought of. Is that a possibility here or is that you just need a lot of data for that type of AI and testing would totally not fit that particular parameter?

No, it's the cutting edge. I think at some point there it might be that way. You know, just because creativity in terms of machine learning is hard. Because right now almost all the algorithms, all this machine learning pieces are all geared towards actually saying, hey, what is going on in the state of the world? And let me replicate it. And that extra creative stuff that humans are just very good at of like the what if questions, those are starting to kind of be tackled today. So I think my concern or my. What I've seen in sort of this, I think is, which is interesting is, you know, we have things like monkey tests, we have chaos monkeys, things like that do something similar. The issue is then I think on that validation aspect of that, right? So how do you validate and say like, say your monkey finds, say this AI system goes and tests all of these crazy permutations of all these test cases, there becomes this, you know, kind of so what aspect? Right? So if it finds an issue where, say on a UI monkey, you know, taps the application icon 20 times, then it crashes. Okay, it's probably a valid bug. I mean, it's replicatable. You can Go ahead and redo it and see it. But then you would look at the logs and say, how many times have we seen this stack trace of a crash? And maybe it's never, maybe it's one time, right? Because people don't tend to click on something 20 times. So there's has to be sort of this balance of validation. And I think that's going to be the main issue. And that's where I think, you know, test engineers who understand the product very well and the end user and they're going to be the ones who would have to kind of sift through and learn from that data. But I think again, it sort of goes back to that whole thing of it's a great tool to help, you know, force multiply that test engineer. Now that test engineer can say with confidence, we've tried, you know, 20,000 permutations on this app. I've looked through the errors and these are the top errors that I think are realistic and kind of bring that in. But I think the realistic aspect, right, is the key point of that sentence, which is, you know, it has to be realistic and applied to that person or that application or that company. And those tend to be very different from company to company to company. You know, I think what if you look at, say what Lyft cares about, it might not be the same as what Uber cares about. And you know, there's obviously some things that they both care about, like making sure you can order a ride, but then there's certain just aspects of, you know, they got in trouble for X or Y and they, you know, need to make sure that it doesn't happen again. So I kind of have always like to say that there's a couple of different types of tests. One is, you know, your core user flow tests, you know, what are your users doing today? And then there's the, you know, CYA tests where it's like, hey, maybe upper management goes to jail if you screw this up. You know, things like we always joked, but it was a Korean age verification. So if you do purchases in Korea, you have to verify that the person is, I think over 16 or 18 years old to do an in app purchase. And so if you don't do that, you're breaking Korean law. And so it's like, hey, we always made sure to have that test even though, you know, that code doesn't ever really get touched. We want to make sure that that doesn't happen because we liked our managers and we didn't want them to go to jail. So but those are things that it's industry slash company specific. And that's again where I think having those testers and those automation engineers be able to leverage and use that tool will be that, that key benefit.

Nice. So if someone wants to get started with a smart driver, what do they have to do?

So we have documentation and so you can kind of check it out, but essentially it's you PIP install or NPM install or maven install it and then you add it to your existing test automation and then you sign up for an account, get an API key, add two lines to your existing test script and then you should be off to the races.

And currently it is, there's no charge as of now. Correct?

Correct. So we're looking into what maybe makes sense, if anything, for charging. We're big fans of open source. We have open sourced our actual SDKs. We're looking into open sourcing our backend so someone can bring this up on their own. It's just, you know, right now we're iterating pretty quickly and so we want to make sure that we have good user experience in terms of bringing up their backends themselves. We're also looking at just rolling this out into the cloud marketplaces so then you don't have to worry about bringing it up yourself. You can just deploy an instance through, you know, the AWS or G cloud marketplace and it'll have the entire backend for you. And so no data would leave your company system. It's just right now it's, it's early days and so we're working on it. Our main goal right now is really just to engage with the community and say, hey, how can we help you? We got YC funding, which is great and so we want to just find ways that we can help the community and, and sort of in the same Google approach, we'll figure out, you know, if and how we can sort of monetize it there. But we're going to keep open source as you know, one of our top priorities because we're ultimately at the end of the day mission driven on this. And so hey, if someone else can, can figure out how to monetize this and then it moves the industry forward, then hey, I would still call that a success. Our investors might not, but I would. So, you know, again, I'm happy to engage, happy to hop on a call with anyone and discuss because I think, you know, this doesn't get discussed enough, which is why, you know, thank you Joe for doing this. And there's a few other folks in the community, but it's the sharing of ideas is I think what helps elevate and get us to that next level and get to these next steps in the community. Just what's the next sort of cool thing? What's going to help all of us in our lives and not have to maintain these tests all the time?

Absolutely. I think sometimes people shut it down. AI is not real and there's no such thing as automated testing, manual testing, and it just gets all kind of weird at that point. You can't even have a discussion. So thank you for coming on and sharing more about it because I think the community needs to know because I don't think you can just ignore it. I know back in the day when I was doing automated testing, people say automated testing can't help and all of a sudden they came along. And I think it's the same thing with AI. For some reason there's resistance. But over time I think it's evitable that it's going to really become part of the every tool probably, I would.

Assume, I tend to think so. I mean, I think, you know, in general people tend to have different mentalities and different ways they view things through their personal lenses. And you know, I don't think anyone is right or wrong. I think it's the right answer. Tends to be a blend. Right. I don't think everything should be 100% automated. You know, I think that there's some stuff that a human eye is really great for, you know, being able to look at a product, look at a user scenario, user flow from all of your experience of understanding your customers is one of the beautiful things about test engineers. They can, they really are the advocates for the customer. And so, you know, that is really hard to automate and simulate. And so, you know, I think you need to have a very well and good balanced blend of those. And you know, eventually, you know, I think the sort of long term approach that some people would think that AI is going is, you know, you could start to just build giant AI robots of, you know, these various people and personalities. So at some point maybe there's going to be a Joe robot, maybe there's going to be a Jason Arvin or a Michael Bolton robot, right. That can kind of just go in and you can, you know, download an AI brain of my Michael Bolton or Jason Arvin and they'll go ahead and test you up like how they would. And you know, that would be, I think, a very interesting future. And I'd love to kind of see that and just play with it. Seriously.

Exactly. You've already started.

All right. Okay Chris, before we go is there one piece of actual advice you can give to I want to help with their automation AI testing efforts and what's the best way to find contact you or learn more about DevTools AI?

Yeah so if you check out our website DevTools AI or send me an email Chris Dev tools or we also got the domain because it's hard to remember of get devtools and aidevtools.com so. So yeah feel free to reach out. We also have a Discord community. Discord's a free Slack alternative and so uh, we have a community there. But I think though in general just AI in general for anyone who wants to start I think it's the barrier to entry in the last couple of years has just been dropped so low that anyone can start to play with these things. So I think there's, it's made by google.com or something like that made by Google but you can start to basically just build AI based models without any sort of expertise or training and you basically just say here's a picture of you know, three pictures of dogs, here's three pictures of cats and it can build an ML model for you that can look at and determine if it's a cat or dog. Right. So I think if you want to play with it you can start to play with those today. I think there's a lot of open source and free tools out there as well that you can check out and there's I think also the AI for software testing community that you know is focused exclusively on this and sends out emails and so that's a good, good resource as well. But I think just being open minded and trying it, I think there is a lot of AI that's out there and some of it may not apply to your particular need or might not work for your particular use case and some of it might work really well. And so I think you know worth at least just honing a craft and trying things out and you know that continued self development is always useful and so trying to find some time to just play with both these tools and there's a lot of them out there, you know not just ours but a lot of great tools companies and people in the industry trying to help solve this problem.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode head on over to test guild.com a 410 and while you're there make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show, and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.