No B.S Guide to AI In Test Automation with Tariq King

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating automation awesomeness.

If you want to really know the truth about AI and automation with no bs, listen all the way to the end. Because we have the king of AI and testing joining us, Tyreek King, to talk all about his journey in AI automation awesomeness as well as his new role at epom. If you don't know. Tyreek King is the Chief Executive Officer and head of Test IO, a EPOM system company. He has over 15 years of experience in software engineering and testing and has formerly held positions as VP of Product Services, Chief Scientist, Head of Quality, Director of Quality, Software Engineering Manager, Principal Architect. He knows it all. He also has a PhD and Ms. Degree in computer Science from Florida International University. So he really knows his stuff. He's been talking about artificial intelligence forever before pretty much anyone I know. So he is the expert. So if you want to get the lowdown on really what AI can do, what the hype is, what it's not, you want to make sure to listen all the way to the end. Don't want to miss it. Check it out.

This episode of the Test Guild Automation Podcast is sponsored by the Test Guild. Test Guild offers amazing partnership plans that cater to your brand awareness, lead generation and thought leadership goals. Get your products and services in front of your ideal target audience. Our satisfied clients rave about the results they've seen for partnering with us, from boosted event attendance to impressive roi. Visit our website and let's talk about how Test Guild can take your brand to the next level. Head on over to test guild.info and let's talk.

Hey Tariq. Welcome to the Guild.

Hey Joel. Thank you for having me, man. It's good to see you.

Good to see you. Awesome to have you. I just realized I'm almost up to 500 episodes and I've never had you on a podcast. We've spoken on multiple events. I've had you on multiple webinars, multiple conferences. So it's great to have you. Before we jump into it, is anything I missed in your bio that you want the Guild to know more about?

No, man. I mean, I think the biggest thing for me is just a passion for quality software engineering and testing. Right. Like, and a big focus on community. So, like, as you know, I'm a big fan of the Guild. You know, I joined the Slack Channels for different communities and try to like just keep in tune. So for me, like that's a big part of my bio as well. Like just being very community driven person.

Love it. So you have a PhD and an Ms. And computer science. What got you into testing though? Why are you passionate about testing?

Yeah, so I mean that goes back to like I was fortunate enough to go to Florida Tech and at Florida Tech we actually had two very great minds in testing, James Whitaker and Ken Kaner. And I had taken a software engineering course with James, which was like my kind of first college software engineering course. It was very different from just a programming course. And so I'd done well in that course with James and I thought he was very interesting. So I was like, well let me just take this testing course and boy was I in for a treat. Like James used to kind of come in and show where like different vulnerabilities were in Windows or in Office. He was doing some work with Microsoft at the time on Office 2003 I think, or Office 2000. Trying to remember the timing there. But somewhere in between that period and I was just fascinated by like, hey, you know what, building software and thinking about construction, interesting problem, testing it and seeing that it was actually working like harder problem, more fascinating problem. And I kind of just fell in love from that point.

Love it. So then what led you to AI? As I mentioned, you're one of the first people I've known that actually started talking about AI and testing way before, you know, ChatGPT4 came out and all that. So how'd you get into AI and testing then?

Yeah, so that would be like a couple years forward from, you know, my Florida tech days and I decided to go and do the PhD. And at that time a lot of the companies from Microsoft to IBM, they had different initiatives talking about the future of systems, right? So IBM had their autonomic computing, Microsoft had their dynamic systems initiative. HP had like their adaptive systems as well, like I believe. So all of these things were trying to portray a future of software where like systems were able to, you know, respond to changing user needs or changing requirements or changes in the environment. And they were talking about systems being self healing and self protecting and so on and so forth. And in that, because I had really like a passion for testing, I was like, hey, we're talking about all of this stuff, but no one is saying that we need a self testing capability. How can we be this dynamic and this adaptive if we're not actually having testing as a part of that, right? So that's kind of where it started for me by just thinking about systems becoming more dynamic. Of course, like the epitome today of dynamic systems is artificial intelligence, which is enabling systems to learn and respond and change and now even generate things at runtime. And so it was kind of a natural bridge from like, hey, AI is actually enabling systems to just become this next generation that we dreamed about. So how can we make a difference?

Absolutely. And you talked about AI testing systems almost self. Self driven in your session. I think you did at a AI guild summit, like 2018. You're also one of the first that talked about self healing. Is the self healing you talked about then, I don't know if you remember, is what it turned out to be now with all these vendors coming out, like, how did you know? Like, I don't know how you have the pulse on it before it actually happens. Yeah.

So part of this was like I was talking about self healing and this, we call them self star characteristics. Right. Self configuring, self protecting, way before it actually became a thing in our space. Right. It was like, look for any system, like how can it actually become more adaptive in a way that as it is starting to fail or even after it has failed, how can it actually recover? And that was the general problem that I think people were describing back in those days. Now fast forward to like looking at the automation space, like what is the thing that is causing people the most headaches that is failing all the time? Well, it turns out like all of these selectors and hard coded ways of defining automated tests have been a pain point because teams are spending all this time churning through flaky tests and trying to make them better. And so it kind of felt natural. Like if we have this technology to help make tests more stable, that's going to be very successful because that's the biggest problem that teams are having this. So even from back then I kind of knew that it was going to go in that direction because you just have to look at the biggest pain point and the technology that will enable it to kind of become a little bit better.

Awesome. So, you know, we've been talking about this for years and years and all son of chatgpt Generative AI really exploded probably the beginning of last year, maybe a little bit before that. Did you see that coming and were you surprised by how fast, you know, when it did come, all of a sudden it's like bam.

Yeah.

Almost game changing.

Yeah. No, I didn't. When I say didn't see it coming, I didn't see, so we had been for years, like, my work back at even test AI, one of the things that Jason Arwen and myself, like, had found ourselves doing along the way, of course, is educating folks about AI and AI for automation. But even in, like, you know, client calls, educating them about, like, what is possible and what is not possible. And most of the time you go in front of like a prospect and they're like, well, you know, these are all engineers. They're like, I want to see the AI, show me the AI, show me where it is, blah, blah. And so we realized, like, from early, like, we couldn't just come and just show them a capability. Right. Even though it was like, great, like, oh, wow, you can do this, that's cool. But like, how did you do it? Right. I think that what we kind of needed and ChatGPT kind of brought that to the realization is that we needed AI to actually be more believable. Like, we needed AI to be able to be in the hands of everyone so that they could get over that curiosity of playing with it and doing, you know, whatever probing they wanted to do. And so I didn't see that coming, but I was very glad to see it coming in terms of the GPT technology. Like, we'd been using that for a while from GPT3, so, you know, the leaps that we see with GPT4, sure, we saw that aspect of it coming, but I just didn't see, like, the whole crave and finally everyone saying, like, yeah, we now believe you that AI is here. Right. And that was refreshing.

Nice. I also was surprised by. I've seen some studies of it, the adoption of AI with testing activities. I think Lambda Test did a study and it was like they did a survey, 76% of people said that they were already using things like generative AI and ChatGPT to help them with the automation. I also did a webinar. We had over 600 people register and we did a survey, same thing. It came out to be like 78% of them already utilizing it, which I was surprised by because it seems like testers seem to be very reluctant or kind of skeptical of this stuff. So are you surprised by how much testers have actually. Some testers have actually embraced AI and started to mess around with it to see what it could do.

Yeah, I mean, to be frank, like, I'm actually just elated that this has happened. Surprised probably a little bit, because as you said, like, most people have been skeptical for so long. But I think that again, because I think for you know, testers to believe they had to be able to experience it and feel it and touch it and go about doing it. So I think because now they've, they've been able to explore it themselves and see it like, you know, that they are now wanting to be part of that whole revolution. So it is surprising that there's so much adoption. But in a way it's like, man, it's about time. Like, you know, and again, I don't expect everyone to be a super fan of all of this. I still expect that there will be skeptics. But I think one thing that's undeniable is that AI is enabling us to do stuff with computers that we problems that we couldn't previously solve with computers. So that's good.

Absolutely. So how do you fight against the hype then? You know, you've been talking about forever, you're a big believer in AI, yet there are probably legitimate concerns that, well, you know, this is probably a lot of hype. A lot of companies, probably everyone's talking about, you know, we're near the hype curve, all of a sudden it's going to be a crash and all of a sudden I'm going to be, maybe it really can't do what some of these companies are saying it can do.

Yeah, fighting the hype is really about keeping it real. As simple as that. I remember at one point, I won't necessarily call names, but Paul Holland, I had him in a course of mine, a CAST conference for the association of Software Testing. And when I joined that, I know, like I was the only speaker, tutorial speaker that year. And I could see on Paul's face already, even before I started, that like he was a skeptic and he was very vocal. But I always try to remind people, even as I go through and I talk about what are the capabilities and also keep a balance on what are the limitations and keep that grounded. Right? Because at the end of the day, if I can use this technology to do some things and it gives me a boost and I keep it real about what are some of the trade offs, that becomes very different than saying like, hey, this is the, you know, silver bullet here, or this is the holy grail of software, everyone knows that that is not necessarily the truth in the current state of the art. Maybe someday it might be, but maybe it might not be. So just keep it real and keep it there. And I remember at the end of that conference, Paul was like, man, I thought this was going to be totally one way. You convinced me like, you're legit. And to me, that was like a big thing because I know that that's the mindset that I feel testers and posters should have to fight, like the hype curve, because it is very strong when you go in front of some vendors, you know.

Absolutely. So how do you see AI then in testing? Do you see. As a lot of people, you probably get asked this question all the time, am I going to be replaced? Obviously, we know the answer is no, but I'm curious to get your take on it. A lot of people, I like calling it like an assistant or a peer programming with AI just to kind of bounce ideas off of and not necessarily take it as gospel truth, but kind of generate bunch of eight ideas that you then could put into place and determine whether or not it's feasible.

Yeah, I actually have the position, Joel, that is a little bit of all of the above. Like, I really do feel that the technology is solid enough that it will not necessarily replace some people, but it will replace some functions. And I mean, historically, if you look at, you know, any technology and the evolution that we've had in technology, there's been technologies that have completely made some things like, obsolete. Right. There's others that have made it kind of like, not so obsolete. Like, we still have ATMs, right. We still have tellers in banks, but there's less of them. And maybe there's more folks actually, you know, sitting down with you talking about your loan. Right. So the automated teller didn't necessarily replace the function of, you know, the human teller, but it did transform the way that they operate. And so I think that there's going to be. But if I need to get cash, do I need a human to hand me a set of cash? Like, no. Like this is. I would hope not. But so I think that some of those functions in a similar manner will be replaced with this level of automation. Others will, you know, maybe be a hybrid and there will be more of this assistant mode, I think. I mean, I use these technologies all the time as an assistant. As you probably know, Joe, these are great assistants, especially when it comes to collating information. Like normally we might have to. Nothing against the Internet, but maybe go do a Google search and pull all these things in right at your fingertips with a few prompts, like you have information. And of course, if you're an expert in it, then you have to fact check it still. But it is great. Now, one thing I'll just add to kind of circle this around and this is kind of where My year has been starting off going, human beings are, you know, intelligent beings beyond anything that we see from computing today. And we have to remember that. And we are the ones that created artificial intelligence. We're the ones that are driving and shaping it. And so when we talk about replacement, like, let's not put that at the human level. Let's talk about functions and things and transforming the way that we do or the way that we work. But really what we need for AI to be successful is actually more human insights for artificial intelligence. And I've actually just made that my theme for the year, to kind of bring maybe the human back into AI a little bit more.

How. How do you bring the human back into AI?

Well, if you start to think about it, if we're doing this correctly, which we might not be doing currently, but if we are going to do this in a way that it really benefits, then we would be doing it for humanity. Right. Like, AI should be something that we see as good. It helps us grow, it helps us automate, become more efficient, and so on and so forth. The only way that you can actually make sure that AI serves that purpose for humans is to have humans in the loop throughout this. Right. Meaning humans involved in AI development, whether that be curating training data, whether that be mitigating, like, unwanted bias, like, there's a whole bunch of areas where during development, human in the loop is key. And then even after deployment, because these systems are, again, coming back to our first topic, very adaptive and very dynamic, so they can do these new things at runtime. You kind of need humans in the loop as, you know, even though there are tools to kind of monitor some things, but to review and make sure that these systems, after they're deployed, are not deviating away from, you know, something that would be very useful into something that's either not useful or even potentially harmful. So bringing the human back into AI has to do with that human in the loop throughout the AI life cycle and recognizing that there's ways for us to have humans be a part of both the, you know, development of these systems as well as the continuous validation and verification in the wild of AI.

Hmm. Okay, so how do you get that vision in place, though? Because a lot of times people may just be stuck at. I mean, it's just going to do everything. Let's not even try to insert, you know, ourselves more into it. Yeah, I don't know if that makes sense, but sometimes people are like, all in and just like.

Well, I think, you know, we've seen already that these systems are fallible. Sometimes you for sure if you ask the wrong question, you get the wrong answer. But sometimes you ask the right question and you still get the wrong answer. I think positioning that well is a matter of recognizing that because the, the system is, has the potential or ability to deviate from whatever would be a normal or whatever you expected it or what it was doing before you released it, then you need to make sure that as you think about your overall strategy pre deployment and then post deployment, that human beings actually serve as a good like reviewer or guidance mechanism for what is the North Star like? We always talk about the oracle of systems, right? The test oracle, like system doing the right thing. Well in this case, like we know that it's even very hard for a human being to define what is the right thing for sure. Like AI in itself definitely will go down different directions and so it doesn't know the right thing. And so to me the answer is always like somewhere in the middle and making sure that you have the right experts, you have the right mechanisms to keep humans in the loop. Right. And it's actually very much bigger than a technical challenge. It is very much ethical challenge as well that we're trying to face.

Absolutely. And the reason why I bring that up is we had a discussion or I had a discussion with someone about how they want to leverage Genai to test Gen AI and you said, whoa, hold up, you may be going a little bit, little too fast here. Can you maybe give us some insight on, especially because they were applying it to healthcare, why that may not be the best idea right now?

Yeah, when you think about these domains, I mean if you start thinking about. And again there's levels to everything, maybe there's a great use case where Jenny I to test gen works fine and it's perfectly good. I would say though that like again, when you start thinking about testing, testing is all about risk. It always has been. Right. And it's about providing information about that risk, making sure people can make informed decisions. This may sound very familiar to bit of folks. Right. But the key here is, is that at this point, one thing that is for certain, like when you talk about testing, there is an element of testing that requires us to have very sound judgment and critical thinking. The very fact that generative AI can hallucinate and actually in its hallucinations be very, very convincing that it is correct. And even to the point to convince a human that yeah, that sounds plausible, I don't even need to check it. It sounded so believable for Me, that's the first point of the matter to say like, hey, you can't trust this in this form. Right. At least in this current state. And therefore if you can't trust it, how can you use it to test anything? That is kind of like the first decomposition of like testing. But again also because of the domain that was mentioned, it changes. Right. Like if it's healthcare or anything that does with human well being is like, okay, like let's take some extra precautions around this particular domain.

Absolutely. So what's the next stage then? You know, Gemini came up with something Google Gemini released was a modular AI. Something where it can like not only use language, but it's use a visual and audio and it could use that for AI.

Yeah, multimodal.

Multimodal, that's it, yeah, yeah. So do you see it then where automation then is not even coding at all. It's just using images at some point and it's smart enough to, I mean, I know I don't want to freak people up, but I could see like that is the way to go. I think eventually if it gets that, you know, use an image to just drive everything, you don't need to worry about what platform, what os, what device, what code.

Yeah.

Is that too crazy?

Yeah, no, it's not too crazy. I mean you, I think you would know. Like I've been a big fan of like computer vision and like the thing is now that like bringing these two worlds together is what is extremely powerful. Like we've always had computer vision actually even before we've had a lot of these advancements in the NLP and with generative AI, like we've been able to train models to detect images on the screen and classify them and so on and so forth. But you know, now with not just the ability to kind of see, but also to interpret and even turn some of what you're seeing on the screen, you know, and kind of bridge, you know, the natural language with that. It is very powerful. Right. Like if I can get even a description of what is on the screen and I can now communicate with the system using natural language to say like what I need to do. There's your testing, there's your intent, there's your verification, there's a lot like it does kind of feel like, especially for UI level automation, now we're agnostic of any device or anything. Like you literally can test anything with a screen. And these are things that like I think we've been dreaming of and that were even before their time and now it Seems like almost like silly, easy to be able to build these systems if we have AI technology like this.

And I think that brings it back to your point, making it human centric. Because if it gets to that point, you still need a human to say, okay, it did automate this, but is it really doing it? Is this what a user really is going to do? You still need that intelligence to verify that it's doing what it's supposed to be doing.

Yeah, see, like, you're right. And it goes back to like a very simple testing concept, right? The idea of validation versus verification. You can verify things all day long, meaning that they conform to some specification that was defined. But how do you know that that specification is indeed what the user wanted? That is something that comes from a human. And, you know, I've been kind of thinking over like the last year, how do we like get this human engagement back in? And it's really about pointing to where the value of the human would be. And I feel like that is on the things that the machine just cannot know and it does not know. And that is like thinking about real users, right, and how it's impacting them, how it's making them feel, how the software is either conforming to their needs or not. And then looking at that real user feedback as a key part of making AI better and feeding that also into the training process. Right. How do we build the right thing to begin with and that sort of stuff.

Love it. So, Tariq, what really piqued my interest was you posted something on LinkedIn. I think you got a new gig. Well, it's same company, but it's a new role. It sounds like CEO of a head of Test IO. Yes, I think. What's that all about? That sounded cool. I'm like, whoa.

Yeah. So that is part of parcel of like what we've been talking about a bit. So Test IO is a crowdsource testing company originally, but a lot of, I guess the innovations as well is taking that crowdsource model and applying it to more than just like manual or exploratory testing. But even thinking about, can we crowdsource test automation? Can we crowdsource engineering work? What about specialized areas of testing like localization testing, which may, you know, leverage something like an AI platform to do translation, but then maybe involve human experts to do that final validation and verification. So part of this is kind of again, bringing together the human and the artificial intelligence, if you want to put it together. Because if we are building the future of systems, then human in the loop where does that. One of those models for sourcing those humans could be definitely a crowdsourced model when you think about a global community. But one last thing on that is also, like, while we normally think of the crowd as these real users or people living across the globe, another general kind of term is more community. Right. And as I mentioned in the beginning, like, I love communities and a community of professional testers. Like, if I can get Joel Colantonio testing my app and giving him his insights on that, then that is terrific because now I have a professional that really knows that testing mindset, and that might match my needs for a particular project. But if I need a real human, that is, like, not that you're not human, Joe.

Right.

But that represents, like, my end user because they use my product and they own it, then I can also source and match that. So whether professional or whether you're, you know, actual user that fits a particular project, like, getting the human involved is all part of, like, this journey with Tesla.

Love it. How do you drive your vision, though? Like, how do you, like, like I said, you're able to predict things. How do you know? Like, okay, I'm putting my resources behind this to make, because I know this is going to work. Seems like a tough thing.

Yeah, no, it is a tough thing. I think for me, like, I'm a very introspective person all the time. So normally I think about, for example, what I'm going to focus on for the new year. Like, a lot of people make resolutions. I was actually talking about this. A lot of people make resolutions. I don't make resolutions because resolutions actually get broken by, like, the second week in January. But instead, like, I really do think, okay, look, I need to start thinking about, like, what is something lasting this year that will be meaningful that, you know, we can use moving forward? Because it's always about building on all of the knowledge, all of the things that we've done from last year. And so it kind of feels like, yeah, like looking into the future a little bit. But sometimes I think it is more of, like, this is where the industry is going, this is where the momentum is. And just thinking about, well, given all that, what is the next step? And like, so for last year, we saw, and like you said, chatgpt, we saw this big boom. Everyone now believe that AI is there, and everyone. The one thing that I did know is everyone is going to be building off of this technology. So we're going to see an influx of these types of systems. And when we do, we also know for sure that they're fallible, they hallucinate. And so how are we really going to control this? Well, in it, it's really two ways. We need automation to actually level up and so we can't slow down. Even using AI for automation, it might not be generative AI that we fully rely on for that, but it might be a combination of things. And then the other thing is, well, humans need to be in the loop because like, even if we do good automated verification, where do we get the validation? And that's kind of how I do it. I just think of like, what's the next step that the industry is likely to take?

Love it. Before we go, is there one piece of non bs actionable advice you can give to someone to help them with the AI automation testing efforts and what's the best way to find or contact you?

Yeah, I would say, you know, again, I'm a firm believer in keeping it real and keeping it practical. All of these tools out there, there's a whole bunch of them, there's obviously a whole bunch of sales pitches as well coming along with these things. Best way to kind of weed through it is get your hands on the technology if it's a free trial or if it's an open source tool like download it, install it, play with it, form your own opinion. Like I'm a big advocate for just thinking for yourself. I never let anyone think for me. It kind of drives my wife a little nuts sometimes. But like you have to think for yourself and inject yourself in there for me. Like best place you can find me is usually on LinkedIn. You search for my name, you'll find me there. Or Twitter or X. Like these are normal channels but also look for me in, you know, the Guild, Slack channel or wherever I like. I'm around right so you can find me.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode hand on over to test guild.com a483 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world. We'd love to have you join the fam@test guilds.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.