Smart TestBrain Automation with James Farrier

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Automation Podcast. We're all getting together to learn more about software testing and automation with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with James Ferrier, founder of appsify about AI and automation. So get ready to discover how to reduce the number of defects created by your team, reduce the time your team takes to find defects, and increase the stability, speed and value of your automation. So listen up.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial, Visit Test Guild.

And click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey James, welcome to the Guild.

Hey Joe. Thanks for having me.

Awesome. So before we get into it, could you just tell us a little bit more about yourself?

Sure. I'm originally from New Zealand and I actually just got my visa to move to the States. So I'm in the process of shifting my entire life over to the US which is exciting. As for my professional career, I went to the University of Auckland and then when I finished my first job morphed from being in Java development to being software testing. And I found that I was enjoying that more than what I was when I was actually developing software full time. So I decided to pretty much stay in that area. So I started moving from job to job and test automation and that's what I've enjoyed throughout my career.

That's awesome. You started off as a developer, then you became a tester, and I think you actually started App Surefy, like in two years. So can you just tell us a little bit more why you started a company around testing and automation?

Sure. For me at least, there have been a lot of developments that I've seen in the development space. A lot of tools supporting developers to make development easier. And at least at my previous companies, we still struggled with our testing. I think that an agile and DevOps testing was almost like the forgotten discipline and it was always the one being squeezed in terms of timeframes. I used to work at a company where we had thousands of mobile automation tests and they just didn't run reliably. We were missing defects and they were getting into production or we were having like these delayed testing cycles and I just thought that there must be a better way. And so I decided to try to start a company to create that.

So James, also I'm curious to know this because I'm going through transition myself. I'm leaving my full time gig to do test guild full time. So what made you make the jump then from developer to tester and then tester from founder of a high tech company?

I guess. So for some reason I've decided to do a whole bunch of things at the same time. Not only did I leave my job, I decided to travel for a year, I got engaged and then I got married. So it was, I don't know, maybe like a quarter life crisis. No, not that I lived that long.

But I think a lot of it is just to do with I thought that this was a good chance in my life where I was stable, I had savings built up and I didn't have anything where I was like.

I had the time and the money to actually give it a shot, which maybe doesn't happen too often.

So of all the things to focus in on with testing, what made you focus in on? I guess what's the sweet spot of App Sureify?

The sweet spot of fshurify is really the automation side. That's what used to be my bread and butter and what I was hired to do. Like I said, I used to work in a company where we had thousands of mobile tests and they would take a long time to run. We had the option of trying to run them on a tool like Source Labs, but sometimes companies don't have that option because they're not able to put them in the cloud. You, you also might not be able to spin up the number of instances that you need to get the results faster. So what I wanted to do is I wanted to build a tool that would get the results back to the developers within that five minute time frame before they get pulled onto something else, before that context switching starts to really ruin their productivity.

Yeah, I know. Also in a lot of new companies that come out, or even older companies that have automation, they're using AI as part of the way that they're marketing it. So is this the same thing with their solution? Do you use any sort of machine learning AI and if so, how do you apply it to testing?

Yeah, so we're definitely different from some of those other companies that you hear about, the testims, the functionaleyes, Mabel Eyes test AI. We're using machine learning in a way that is sort of already being used by some of the big companies, the Googles and the Facebooks. And in fact they've actually written papers about it. So what we're trying to do is we're trying to say for the set of tests, which is the small subset that you should run based on the changes being made. So we're not building the tests for you. What we're doing is we're trying to prioritize your existing test suite, and that's using a classification model. And then we have a couple of other tricks that we do where we try to detect whether a failure has been caused by a real defect or whether it's one of those flaky ones. So we have another classification model there. And then finally we have a little bit of a defect prediction model where it's saying, these are the commits that have come in. What's the risk associated with them?

Use the term classification model twice there. What does that mean, classification model?

So it's really just taking a lot of data through and then trying to work out for each point or for each data point, what should I classify that as? So for an example, the most common one is trying to do that image classification. You've probably seen lots of those where it's like, is this a cat or is this a dog? But what we're trying to do is instead say, is this test going to be valuable or not? Is it going to find a bug or is it not going to find a bug? And then similarly for the commits, whether they're risky or not, is this commit going to contain a bug or not? And so we build out that model based on a large number of features. We do it based on sort of developer tendencies. If you see Joe come in in the morning, and Joe usually doesn't get up before 10am and he suddenly does a commit at 7am, that's the type of feature that we're looking at to say this might be risky, if that makes sense.

Absolutely. So it seems almost like a team tool that can analyze your team and their kind of quirks. And then based on history of maybe you have a developer, like you said, that some reason that checks in code every Friday, and every Friday he breaks something. So maybe you prioritize that as maybe a higher likelihood of causing an issue.

That's pretty much exactly it. We actually often see things like on a Friday, that is when bugs are more likely to be created. And that makes perfect sense. Right? People are trying to get things out before the weekend, or it might be you've got a scrum ceremony on, let's say, the Wednesday, or an Agile ceremony. So that causes your developers to maybe rush to get something in, which again, is more likely to cause bugs. And we're looking at other things. Like I guess what we're trying to do is replicate what a really good tester's instincts are. If they see a commit that's come in and has changed four areas or has changed a file that they know that only Jess is allowed to change, then that's the type of thing where we can say, look, this is risky because that's what, what we've learned.

Awesome. So I speak to a lot of vendors, so I try to compare the different solutions to see if they're the same, similar. So maybe I'm off with this comparison, but I'm just trying to get my head around how you're able to tell with code check ins if, if a bug has been introduced or what test to run. So there's a company called C Lights and I guess what they do is they analyze, you have checking code and because they have a baseline, they're able to know that, you know, this code touches functions A, B and Z. And that test, you know, 1, 2 and 3 cover A, B and Z. So therefore only run those particular tests to cover that particular code because they know exactly what's changed. So how does App sure if I know that when someone checks in code, are you doing something similar like here are the test sets to run against this code or you're just saying here's the code that was checked in and here's the likelihood of having a bug in this particular area?

We're doing both. We're saying not only what is the likelihood of there being a defect, but we're also saying because it changed these areas of the code very similar to the way that C Lights does it, These are the tests that you should run. You're correct. They are, I guess, in a similar space to us for that piece of functionality.

Nice. Again, I speak to a lot of people and some people are very pro. Run your tests against all the things all the time, against all the systems. And even if there's a small check in, I asked why should I run all these tests when I know a lot of noise is going to be generated? Because I know the check in doesn't even touch 98% of the tests you want to run. Why is it better to know what tests to run against what code was checked? How much confidence can you have to say, I know that these tests match with this code check in. And what I have in the confidence of running all your tests to say, okay, here are some. I know some unintended consequences could occur. So therefore I should run my whole regression suite.

Yeah, so I can see what you're getting at there. Essentially what we're trying to do is not provide like 100% confidence. We're trying to provide, you know, that 95%. And instead what we're trying to do is get the benefit to the developers and stop them from doing context switching as soon as a developer starts moving on to another task.

If your tests take an hour to run and it's going to take an hour for the developer to get the feedback that he needs to know if his change is good or not, he's going to move on to something else. Or he's going to go and drink 10 coffees and be way too over caffeinated or he's going to go and potentially start changing something else and be in the middle of a change and then have to go back to what he was previously working on. That's the type of thing that not only destroys productivity, but it's also more likely to cause bugs. So we want to get feedback to the developers as quickly as possible, which has a high degree of accuracy. It might not be perfect because we're not going to run the entire suite, but it's just about getting feedback to him so that he doesn't become stuck on another task, so that he doesn't have to switch between tasks.

It makes perfect sense. And a lot of times if you have longer running test suites, you don't get that feedback in time. This sounds like it's awesome because people again, almost instant feedback then to make sure that that loop is closed sooner rather than later.

Yeah, the whole point is to shift left and actually make, you know, get those results to the developers. Just, just like you're saying.

Very cool. So another thing that seems different about your company, maybe I just missed other ones, is you have copy about manual testing. Like prioritize manual testing on the risk. And usually when people hear AI and automation, they think the company is just pushing automation. So what's this feature of how does this help you prioritize manual testing?

So you can push the manual test to us in the same way that you push the automated tests and we will learn from that. Alternatively, what we do is we actually give you a risk map. So these are the commits that have come in, these are the areas of the application that have changed. And from that that's where you can focus your, your testing efforts, whether it's working out which test cases to run or working out which exploratory scenarios to run. I guess that's all coming back from when I was at a previous company and we were shifting from waterfall to either agile or what some people call a slightly faster waterfall, where we were struggling to work out what to test. We were missing bugs because sometimes developers would say, look, I've only changed this area of the code and then might not fully understand the repercussions of their change. So we're trying to highlight one, the risk of each change and two, what's actually changed for the test team so that they can really make sure that they find those bugs.

What other features does appshurify help you with? Does it help you with flaky test or help you try to make your test a little more reliable or more targeted to. So it's not. So you don't have the flakiness that's normally associated with automated test?

Yep. So we are doing that as well. When results come in, we're essentially categorizing the failure reasons into defects. We're actually automatically raising those defects for you and then we're actually even closing them when they pass. Let's say this developer comes in, he creates a bug, causes five tests to fail. We'll raise a defect for that and then when those five tests pass again, we'll close that defect. And we can do that in Jira, if you're using Jira and we're going to add other systems as well. The reason for doing that is my previous company, we've got all these tests and they're not reliable. They are flaky and people are spending hours going through the results and then even raising bugs and sometimes would get duplicate bugs because the same defect would cause multiple tests to fail. So we're trying to essentially automate that process to reduce the manual load of the, of the automated tests. And then like you said, we are actually trying to automatically detect flaky tests, or flaky defects we call them. So that we're looking at when the test failed, did it fail early on in the cycle? If it fails early, then it's more likely to be flaky. We're looking at the reason why it failed. Is it a assert failure, is it an error? We're looking at the history of that test and then we're also looking at other similar defects to work out is this a flaky defect or not? And then we're classifying them as such. What's interesting about our solution is that we then use this knowledge to determine whether your build should pass or fail or not. Let's say developer one comes in, they create a real bug, some tests fail, we're going to break that build because obviously that's a real bug. Let's say Developer 2 comes in and you've set the build to only fail on new defects. If developer1's defect is still there, we're actually going to pass that build because we already know about that defect, it's been raised in the system and you can actually configure this. Then similarly, let's say Developer 3 comes in and suddenly they run their tests and one of them fails because a mobile phone has an issue or something's gone wrong with your Selenium grid, we're going to detect that as a flaky failure and stop it from breaking your build. Because that was the most frustrating thing for developers at my previous company. It was why they didn't like the automation. Right. It's breaking their build when it's not their fault. So we're trying to prevent that from happening.

Oh. Oh, my gosh. So I work with, I used to work with a company that had eight Sprint teams and they would just ignore the test after a while because they, oh, it's an outstanding test, an outstanding defect. And then when the defect was fixed, they still would ignore it because they would, would still think that the defect was still open and so they would just start ignoring it. So it sounds like this is a great way of keeping the people looking at the dashboard and paying attention to the test because it's handling all that type of information for you. What are you having to remember? Oh, yeah, well, these set of tests have a known defect, so I can ignore them or these are supposed to be fixed, so maybe I should look at them now.

Yep, that's exactly how it works. And I have a very similar story where my previous company, again, we've got so many tests, so many tests are failing per run. We just expected there to be like an 80% pass rate. So people stopped looking at the defects and especially the developers. And then that's when there's actually a new defect and it just gets missed because no one's actually looking at the results. And that's, I think, a lot of the reason why some of the automation projects don't, don't give the value because people actually stop looking at results and then they stop finding the bugs and then people think, well, the automation isn't helping. But really it is finding the bugs. You're just not checking the results.

Absolutely. And also it's not just the developers, testers will ignore it because they're like, I'm not going to triage all these tests when I know that 90% of the time it's not a real issue, it's a known defect. So it also sounds like it'll help with that type of issue as well.

Yep, it's automating that process for the testers. And. And the truth is testers don't have enough time to do that type of thing. We struggle to get time to just do the testing that we need to do. So when something else pops up that's supposed to be automated, it really does need to be automated. Otherwise it's just adding additional load to the team. And this is, at least for now, where I see the biggest benefit of our solution and machine learning. There's almost.

Some people have a negative idea of machine learning for testing, like it's going to take the jobs. The way that we're trying to push it is this is actually just going to give you the time back that you used to have when you were doing longer phase testing and reduce some of that grunt work.

So how long does the tool need to run r to be able to give you these accurate type of statistics? A lot of times people say, oh, you need to run your test like thousands of times before even this data can make sense.

So right at the start, we actually have a mapping process where we can map the tests to the code and that's where essentially you can have a starting point. And that at the very least, is going to reduce the time that it takes your tests to run by a significant amount. And then it's going to learn over time. And the interesting thing is it's not going to learn just from a number of runs. It actually needs to learn from failures. So it needs to see bugs in the code for it to learn from. We're also connecting to your tfect repository, if you allow us to, and the source code, we're actually trying to find bugs from that to then build up the data set. So it's not going to take you thousands of runs. We can actually get this set up in a matter of an hour or so when you're going to start to get value from it.

Cool. So how does someone plug this type of solution into their pipeline?

Are you running things in the cloud as well or are you just analyzing the data and then they can run their tests, whatever infrastructure that they're using or prefer?

Yeah. So we offer three solutions. We'll do an on prem install, but I'm still trying to push people towards the cloud. Which case we've got two other solutions. We offer a hybrid solution and then we offer a full cloud based. The hybrid solution is essentially just a script that connects to your repository which then pushes metadata to us. So you'll be able to see exactly what's being pushed to us and fully. Then you can do a security audit a lot faster than on a full cloud based solution. And in terms of connecting it to a CI CD pipeline, that's actually surprisingly easy. We are just a script that runs over the top of your existing way that you execute tests and then it does API calls out to us. The first one is which test should we run? Then it runs those specific tests. Then the next one is pushing the data back to our system and then the final one is saying should I pass or fail the build based on the results of the tests.

Very cool. So another common trend that people are doing is dashboarding, obviously all kinds of dashboarding for all different people. So you're gathering all this data. Do you have any APIs where people can suck in that information and throw it up into like a.

Electric stack or grafana? Whatever the dashboarding solution is, are they able to consume the data that's being collected with your solution?

So we don't currently offer that because the tests are being run in the same way. They could collect the data in the same way. But we also offer just a dashboard ourselves where we are trying to get that information to them so they don't need to create their own. So we give you that full experience of like an allure reports or extent reports in our system.

So maybe that's functionality that we'll want to add if people are interested in sucking that data out themselves.

What is the solution though is every little bit extra? Like someone could just get smart test Automation and someone could just risk space. Or is this called Test Brain? I guess I'm looking at your copy here. Does the product Test Brain and Test Brain then gives you the functionality of smart test automation, risk based manual testing and risk failure elimination.

Yep. So it's, it's really all in one product at the moment. I don't think we have plans to split it out. This is really just about creating a solution that helps testers in whichever way we feel best. And the truth is most of the work comes from, you know, the data that we're pulling in and we sort of need the same data for all of our bits of functionality. So splitting it out might not be useful, but you can use which parts of it that you want. If you don't have automated tests, the risk analysis is still going to come through. We actually send emails out now and alert you of like, high risk commits or when someone changes something they shouldn't. So all that comes packaged into one. You just choose what you want to use.

What's the dev metrics? Is that the same thing that we talked about earlier with the teams? If someone checked in code on Friday and they do it all the time and break a build, is that what DEVmetrics takes care of?

No.

Dev metrics is something a little bit different. We're still playing around with this idea. We built it. So when I was building this product, one of the first customers that we were working with asked me, how are you determining this risk? How do you know that this is going to be a risky commit? I said, well, we're actually collecting stats about the individual developers. I can tell you that this developer often creates bugs when he changes this area of the code. I can tell you that he often creates bugs at this time of day. They said, well, why aren't you showing this to me? That's how we actually started with Devmetrics. We measure each commit, we try to work out how much risk is associated with it and how long we would expect it to take, and then we can tell you how long each developer took. So we can essentially score developers based on how they change code and actually each area of the repository. So we can say this guy is best at the account section of the app and this guy is best at the transfer section of the app if you're doing banking. And we can tell you who's creating the most defects in those areas. So I always thought that maybe the way to help test as most was to try to reduce the number of bugs that the developers were creating. And this to me seemed like maybe a good solution, if I could tell. And in fact, again, it stems from my previous experience where I knew a developer would create a bug. This one guy, I knew he was going to create a bug on this task. You just had a feeling you'd seen it before so many times. But I didn't really have the stats to back up my thoughts at that time. And this was trying to give actual real statistics around those types of insights so that then maybe either better assignment of devs would actually reduce the number of defects that were created or even increase the speed of development.

So how does this work? Are you instrumenting someone's code or is there an agent running on machines to collect these statistics? Someone has an existing Selenium Automation framework, how would they be able to use your solution?

So, like I said, the way that the Selenium test would run is pretty much the exact same, except it would be doing an API call out to us to say which tests to run. In terms of connecting it to the repository, we would. We're not actually instrumenting. Instead what we're doing is looking at the commits as they come in. We connect to the repository itself, we get all the information about the previous commits, what files they've changed, when they were sent through all the git log information. Then we do a manual mapping. Say, look, these tests map to this area, or this suite of tests maps to this area. And that's where we get started. And then when a future commit comes in, we get essentially the metadata about it, which files were changed, which areas of the code, which dev did it, and then that's what informs us of the tests that we should run. So we're essentially linking test runs to commits.

So, James, who's the target audience for this product? Is it developers, testers? Who should be using this solution? Who's this made for?

I think probably the best way to answer this is actually just saying that we're looking for teams where they either have a large set of tests and it's taking a long period of time. All those tests aren't reliable. So you're getting that dreaded 70% pass rate, 80% pass rate, where the automated tests aren't providing the value.

The other part where it's really useful is where you start to get those larger teams and not everyone has a full understanding of the commits that are coming through, of what the developers are doing and the consequences of it. Where we can actually do that tracking for you, where we can alert you when things are going wrong, where we can tell you, look, this is high risk, where the teams are just either so stretched or so large that they can't get that information themselves. That's much easier when you're a small team, you know the individual devs, where there's no, you know, you've got one Scrum team, one tester and four devs, then you're going to know the risk and the changes that are being made. Whereas as it expands, as teams get bigger, that's when you start to see these problems where people are missing bugs, where developers essentially conflict over each other, and that's where we can be useful.

Okay, James, before we go, is there one piece of actionable advice you can give someone to help improve their automation testing efforts? And let us know the best way to find Contact you or learn more.

About App surefy I've been doing a couple of talks at meetups and conferences and I've at least for me there's these four pillars of test automation. I think that everyone should be involved with the test automation. If it's just a tester's job, then.

It'S always that one step behind the developers. And when the developers are involved, you get all that developer rigor. You get code reviews, the tests are checked in with the developer code, which means you conversion them. So I think that that's probably the most important thing from my perspective, that the tests aren't just owned by the testers, they're owned by the full team. They're visible and you have this developer rigor around them. In terms of contacting me, you can find us at our website. We're appsurify.com I should probably spell that out given how much difficulty people have with our name. That's a P P S U r I f y.com and you can find me at james farrierify.com otherwise we do have Twitter I think, but I'm terrible at social media, so I think that's psurify.

Thank you James for your automation awesomeness. For the extent of value we covered in this episode, head on over to testguild.com alikeautomation a279 and while you're there, make sure to click on the Try it for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. And if you haven't heard already, it's actually not really advertised yet. I did a soft launch of Automation Guild so I'll be doing a big promotion coming up soon. But if you are itching to find out who's speaking so far, easiest way probably is just to go to testguild.com auto a u t o so testguild.com auto so that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating Full Stack Automation Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the git build and continue your testing journey. This has been a Joe Calantonio production.