221: Tester in Your Pocket with Evan Wiley

I just think would be a valuable thing in the future just to have something that would be able to learn from you. And it's, and it's kind of having another would say tester in your pocket, but sort of like another thing to be able to like leverage as you go along.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is.

To help you succeed with test automation, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of Test Talks. In this episode we're going to be test talking with Evan Wiley, a quality engineer for a very large financial company, all about how to get that automation culture into your Sprint teams. So listen up and discover a whole bunch of real world experience around enterprise wide automation testing and in action. You don't want to miss it. Check it out.

Test Talks is sponsored by the fantastic folks at Sauce Labs, the cloud based automated testing platform that eliminates the need to maintain your own selenium grid and test infrastructure. Try it for free today. Visit testalks.com and click on the sign up now link under the homepage sponsor section.

Hey Evan, welcome to Test Talks.

Thank you.

It's great to have you on the show. I know I heard from you from Moshe from Applitools. He was really excited about what you were doing at your current job. So I'd like to talk to you about test automation, AI and anything else that may come up. But before we get into it, could you just tell us a little bit more about yourself?

Sure. So I pretty much started Capital one back in 2013.

Called the TDP program, the technology development program and just started kind of doing manual testing and then we kind of had our Agile journey and started to do ATTD and everything else within Capital One and that kind of got me in the path of learning some Ruby cucumber, doing that within Sprints and then moving into different frameworks within different teams. And then eventually now we just used Protractor Cucumber for some of our testing and just doing some exploratory work around different side of CICD implementations we have and just trying to re revamp some things. You know, that's kind of been my journey is, you know, going from the manual testing to, to automation to you know, learning different frameworks to kind of learning what like teaching other people about it and kind of having other people outside of body engineers just teaching other people how to do test automation. And that kind of, you know, part of my journey was, you know, being the Only quality engineer on team. Sometimes it's teaching other people how to do it. So it's kind of easier to spread the workload and sort of being able to teach people that is, is valuable because then you can kind of learn from them, you know, they learn from you and it's kind of, it makes it more like of a cross functional team.

Awesome. So I love that you're teaching folks. That's one of the things I see as a roadblock to implementing automation is the knowledge, even if they're developers. So any tips on how to teach someone automation or how to get that automation culture into your sprint teams?

Yeah, I did it sort of as like workshops. So I first started with sort of like when I was located in Richmond. Sort of taught people workshops about what are some of the best practices of when you first start. You know, I think when you are talking to different like web developers or whomever you want to put it in terms that they understand. So if it's a UI developer, they would understand some of the concepts around. Okay, this is our DOM HTML page. And this is sort of like the layout of how you would go about looking at it. And when you say different types of tags on the page, like this is an H1 tag, you kind of like go through a flow that they understand and then you tell them how automation fits within that picture. So kind of saying, well, if we have this story coming up on say like I guess make a payment or something that's related to like a story that you're doing, you could explain to them in terms that they'll understand of like the, basically how the workflow goes and then how does automation capture all those business processes in order to best produce the desired outcome that we're looking for? So I would explain it in terms of like everyone on the team would know acceptance criteria. So yeah, I'd first start from the aspect of acceptance criteria and then I would kind of explain the workflow of whatever framework we're using and then tie that back into. How is this atdd? How is this, how are we driving testing forward with having the, the gherkin first and then, and then structuring our tests around that. And I would, you know, in terms of our framework I would, I would say, okay, our gherkin is the top layer acceptance criteria. Given I'm this person when I do this, then this will happen. So you could explain it in terms of the business actions first and you would say, okay, we have methods and different step definitions that tie this all in together to Say, okay, we have this gherkin and it's tied to this action. And I just say all these sub layers are just actions that your browser is taking in the background while you are running these tests. And then once they explain in that you would then go through, in these workshops and go through sample tests that you could create with them. Just create like a small exercise to have them understand how the, how automation works and have them write their own automation tests. And then they would understand how that workflow works for themselves because I think, you know, you would, you can kind of explain it and then you have to let, let people do it themselves because, you know, people I learned by doing. So if you have other people that are willing to just, just do it themselves and that, that'll make it a lot better use of, of the team's time and you, you'll be able to see more throughput with the quality of the repos too. Because from what I've seen is when I've worked with developers and I've gone through these workshops with them, they've actually shown me some things that I didn't know before. Like, you know, we refactored something, make it more cohesive. You know, they come from a different mindset. You know, you always have that prototypical sort of QA hat and dev hat, and then you, when you switch those, I think it makes it an even more interesting picture of how does this play into your automation or the technicalities of how your framework is built? Because they would understand better how the actual your site is built and you, and you know, the test would understand how do we make this better. So if you combine the two, it kinds of makes for a more cohesive framework. So at least from my, from my own experience, it's been beneficial to just to have that. So, and then they kind of grow and appreciate the understanding of like what a, of what a QA person does.

Cool. So, Evan, you did mention that you did learn from them as well. So is there anything you remember of what was refactored that you thought, now I know that that's a better way of doing what I was trying to do.

Yeah. So a simple example would be, say, we incorporated Apple tools into our automation testing. So when I was actually going through different, like our web application, I would say, okay, I visualize blank page for this length and width. So when I created that gherkin, I didn't realize that I could just like plug in the length and width with the actual page and then make that reusable across. So there's just simple things like that that made it a lot easier for me to be able to just plug and play with it. We have RSA also built into our framework. So, like, getting the RSA token when you were going through our test automation flow and doing these API calls to RSA was a little complex. So, like having a developer there that understood RSA and he was actually able to implement that within the automation framework because it was something that unless it was the developer, it would be. It wouldn't have the same context behind how is the RSA plugged in? Like, how is this RSA token plugged into our API calls? So there's just, just like some wins we had along the way to say, okay, this was something that they had implemented with API calls and then also like, just some refactoring. Oh, and also like, we implemented a configuration, you know, when you run your browsers in the automation framework, to have multiple browsers all combined into one configuration file and just have all, like, have it in parallel, but also run in different browsers at the same time, that was a win as well, because I typically prototypically thought of configuration files as just running Chrome, running Firefox, running ie, but you can actually, like, combine different browsers and with multiple threads into one configuration file. So that was just something that I had learned along the way from them. And that really benefited my understanding of, like, looking at it from a different perspective, of saying, okay, well, if you take someone into a new skill set, then they'll probably have a fresh understanding of that base of knowledge and possibly reimagine what you're thinking about in terms of the actual framework and how it's implemented. So, yeah, so those are just a couple off the top of my head. Just this, this one that I was thinking about.

Very cool. So you didn't mention applitools. So what kind of visual validation are you doing?

Good question. We basically had, like, we had our web application without going into physics, but, like, just going into, like, there's different browser windows and different layouts. So when we were actually testing, there was a story where we had a responsive story. So it's basically different length and width. So when we went into it with tablets, say if we had someone with a Chrome browser running on their MacBook, we actually wanted to test that if it was responsive that we, we would redirect them to our mobile app. So there was a separate team to do the mobile app and we were doing the web app. So we had the Chrome open and then navigated to the URL in a full page. It would just show the homepage we had the login and everything, and then it would show the dashboard afterwards. So when we actually resized the page, I believe to a certain length and width, we said, please use the Intellix web browser, please use the desktop specific sizes. And we said that mobile and tablet is supported by the lab that we're supporting. That that was kind of like things that we did to refactor some of the tests we were doing. So that, that way if they went to that browser size that they would be redirected to that. So basically my Apple Tools would go through that whole page and I actually, I found some defects through it because when I implemented applitools within our own mobile web application, actually one of the stories, the responsive story that we had, I realized that at this certain pixel length and width that it was not showing responsive that the actual design we wanted for redirecting to an actual web browser, so it wasn't showing the design that the UI designers had wanted for us to display on that length and width in the mobile browser. So I decided to report that defect because it was a defect. It also found this other defect where I was in the middle of transitioning to a new team. So I decided before I left to do just some hardening regression and everything. And then when I was running Internet Explorer, I believe in Apple Tools, the regression failed. I went back to check, check on it. It was actually not even displaying the webpage in Internet Explorer. So it was just like these different defects that found through Apple Tools that was just very interesting. And it's just another good tool to have the bag because, you know, you never know when you'll need to use it. And I find that, you know, the best UI defects are the ones I found through that because it was something that you just can continuously test for. And sometimes it's hard to just eyeball certain things and say, okay, is it, does this match what I, what my wireframes are and everything? And, and due to all my efforts with that, like a lot of our company has adopted like Apple Tools as like a visual automation tool and we have like a Slack channel to support it. So it's actually been really useful, you know, for people, I believe, you know, going forward just because it's, it's something that was, was just helpful for me and I believe that it'll be helpful for more teams going forward.

Cool. So speaking of tools, who chooses your tech stack and is it a shared framework they use across the organization or is it just team by team depending on what they want?

Yeah, the tech stack is good questions. I would say it's more of a. It depends on what it is. So I think that you know, based upon line of business and different needs for different teams, it can vary. I know that a lot of influence comes from kind of like a tech lead side. So it really just depends on your own organization and how you kind of like structure different projects around different, different aspects. So at least with our team I know like typically I've seen it balance around kind of like architecture requirements and how do we want to design this and how does the best design go around? Each basically stack. So I know a lot of things we use are like postgres and different things though a lot of our stack uses whole the mean stack and just this different things that were. I haven't been in those conversations but I know that a lot of that drives from kind of architecture requirements. What's there, what's existing within Capital one? Because sometimes when new applications are built or new things business requirements form, we say okay, what's existing? Like oh, we're connected to TCIs or we're connected to this. And we could say okay, how do we hook into existing applications and then just being able to support that within our own organization. So that was kind of just thinking of probably the best architect here would be one that would support what we have as an existing application and everything.

Cool. So usually before I interview someone I kind of mini stalk them. So I went to a LinkedIn profile and it looks like you do a lot of different things. You're always experimenting. Like if you look at the list of things you've done, it's very diverse. So one thing that popped out was Alexa skills. So I think audio is actually going to be a big thing that folks are going to have to deal with in the future with these type of devices. Google Home and Alexa. So have you done any sort of testing with audio and if so, how did you do it?

So I haven't necessarily done any testing with it. It was sort of like one of those innovation sprints where we had a lot of some bandwidth. So we had a high five application where we were kind of congratulating people for the work they were doing and it was different like things people could put in comments to say, oh thank you so and so for helping with, you know, this test data that I needed for this print or this dependency or just complimenting people in general like you know about like oh, you've been a great product owner, so. And so thank you for all you do. So this, this hi Fi application allowed us to look at integrating Alexa with these standups. So like when people were giving their stand up after their stand up, they'd say, alexa, give me the high fives for today. And it would say, okay, these were the people that were congratulated today. And it would read out the different high fives that those people had for a certain day. Yeah, so we had different like skill sets built around that. At first it was more. The interesting part is that we had a lot of issues running into sort of, I wouldn't say security but like I would say probably more or less like being able to implement a more secure way of doing the lambda because a lot of lambda calls are not within like are not secured within VPNs. So a lot of like say if you were to create a lambda now someone, if it's public, so someone from Russia could like use that same lambda call and use it for something else. So it's like one of those learnings where we, where we had built something and then realized that oh well, we can't really use it right now because of security issues. But you know, I think that it's been a good experiment for us and I know a lot of other public applications at Capital One have used Alexa for different things and different public applications that actually are using it. I've learned that we do have like our own integrated voice recognition like test scene where you know, I was just sitting in on one of our. We have kind of like an automation guild or internally. So we were just learning how, okay, we have this framework and this, I think it's called framework called IDR and uses kind of pre. Records everything and does different, you can do set different permutations with it with a prerecorded voice and just say, okay, this is one flow and it'll test these different flows and then see if there's like defects but then the voice activation flow for that, for that particular user.

So Evan, we finally get to the topic I think you wanted to talk about which was innovation with AI test automation. Why do you think, what are your thoughts around AI automation and innovation around that?

I feel like there's a lot of, there's a lot of room, a lot of different things that I've been reading and I know you've posted some things. I was going to try to explore some things within my own team and see what, what is, is there anything we can hook into to have that Running as part of our own things that we do within our own group. Part of it is, I think that there is more room to grow with having like sort of a learning pattern of being able to teach like an AI about your application as you go along and then having it learn different actions based upon that. So I was just thinking about it in terms of the perspective that maybe it would be something in the future where you would just say, okay, well I have like have it on record at this moment and I've given it this actions and it's learned this thing and then I press up and then after it learned that it'll see other things on the page. And then being able to like experiment with that and run its own test edge cases around what you've given it, I just think would be a valuable thing in the future just to have something that would be able to learn from you. And it's kind of having another would say tester in your pocket, but sort of like another thing to be able to like leverage as you go along. Because you could possibly always have it running in the background in your production environment just to say, okay, maybe in a sandbox production environment just to say, okay, it's there and it's. And it's. And it's constantly validating for me, you know, so like if something breaks, then it'll just like about it and sort of not, not necessarily a test spot, but like something that's kind of like that you don't have that. That you can feel a bit better that something else is there as well. Say if you're. And it's learned everything from your app during your application life cycle. So it's. So it's kind of like if even if some resources on your team leave or anything like that, you'd still have. Or maybe it's like this one app is not supported anymore. So you could just. But you could always have the test bot running in there to say, okay, this is okay. And we like for existing users this, it's still working the way we want to and it's learned through the inputs of the other testers of like how to continuously test for units. Almost like a way for you to continuously test without actually having to. To test it.

Cool. So once you actually implement something, I'd love to get you back on the show to talk about it because I think it is.

Yeah, yeah, yeah, I just, yeah, I've just been learning about it. But like just I, I definitely was starting to dig into it and, and baking in as part of our innovation for like different PIs and just on my own learning, just to like learn it on my own.

So do, do you think it's the real deal based on what you've seen so far?

Yeah, I think it is. I mean, well, so I haven't like I need to invest more time just to like be able to completely know the scope of what's there and what exists. But like I've just been doing my own like cursory research about it and it seems like there's a lot of data out there that point to that it's possible. But like I feel like there's, I think there's still some evolution that needs to happen there for it to be fully usable yet. But I think that there's definitely different parts to it that could be replicated and things like that. But I think just from the different languages and things that I've seen with AI so far, I don't see it as yet as something that can be done. But I definitely just like I say if you keep learning about it, who knows what will come out of it. Sometimes it'll be good and sometimes, you know, even if you like have some few mistakes along the way, you can like learn from that and grow from that. So I think that there's a lot of different opportunities there. I would say even if it's not what I'm expecting, maybe it'll turn out to be a different way. So I think just if I keep learning and keep taking different courses. I was planning to take a Udacity course just to kind of get my feet wet with, with learning it. So I was at first trying to, to learn some front end development so I could understand kind of the, the development cycle a little bit better. So like the part of it was just, I think just continuously learning that app that of the subject matter. But just from my own like research I think that there's, there's definitely possibilities on the way. But I think if I, if you keep a breath, keep like learning it as you go along, then it'll like, once it kind of seems like it's a possibility at some point, then it'll just be a lot easier to see how that would work, certain use cases and so forth.

Are there any conferences or resources you recommend for folks that want to learn more about test automation?

Yeah, for sure. I definitely feel like the saucecon has been good and I know that you support a lot of kind of different conferences that are good for that as well. So I think there's a lot of good resources out there. There's the online test conferences that are good as a great learning source for different material coming up. I think there's CA was a good one. I think it's called California. I have to look up the name. But that was one where it was like kind of like a virtual conference and you could go to different booths and hear people talk about different subject matters and topics. So I definitely feel like there's a lot of good resources out there for it.

Kevin, before we go, is there one piece of actionable advice you can give someone to improve their test automation efforts and let us know the best way to find or contact you.

Yeah, for sure. So the best resource. I've actually been building up my own website for learning about just to impute my own knowledge about what I know about quality engineering then try to teach that to other people. So that's coming soon. So don't have a site on that yet. But. But I've just. And also it's been helpful for me just to have my own knowledge of develop my own site just to have that awareness of how. How it works that and I think definitely your site Joe has a lot of information on that. I know Dave Hefner was a great source as well. He's actually been like I get his emails periodically.

For the test automation. So that was in his guidebook for Selenium is pretty good. And I know Jeff Morgan. He actually came to teach Capital One when we were doing our ATDD change. He actually taught us more of Cucumber Ruby and he has his own book as well. So I can provide you with that information after the call just to kind of say okay this is kind of the resources I use. But. But those I say those three were kind of the resources I went to. And then of course there's like Google Community groups, there's Protractor group if you're in Protractor. And I would say just like there's different like resources at your own work. Like there's. We have our own Slack channels Protractor. We have our own Slack channels for like Apple tools. We have our own Slack channels for our automation guild. So I think there's definitely resources online and definitely resources within your own group or company or just in general if you look at stack overflow.

So there's different. That's probably the ones that I go to in terms of learning an educational is probably really good as well.

Awesome. And Evan, the best way to contact you. Do you have a Twitter handle?

Yeah, yeah. So it's called Test Excellence. And that's also I have a Facebook just Evan Wiley.

Thank you Evan for your automation awesomeness. For links to everything of value we covered in this episode, head on over to test talks.com221 and while you're there, make sure they click on the sign up for a free trial link under the Exclusive Sponsor section to learn all about Sauce Labs awesome products and services. So that's it for this episode of Test Talks. I'm Joe. My mission is to help you succeed with creating automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Talks podcast.

Head on over to www.testtalks.com for full show notes, amazing blog articles, and other automation awesome.