How to Choose a Test Tool with Darrel Farris

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Daryl Ferris, all about test tooling and other automation and AI topics. Darrell has worked in the software testing industry for 20 years. He's focused in on areas of quality and test. Currently a test automation solution architect at Mable and he helps teams integrate Mable's intelligent testing platform into the delivery workflows to enable better test creation, monitoring and issue discovery. Really excited to talk to him. I love talking to folks that have real world experience as well as vendor experience because you get a really good overview of what's going on in industry. You don't want to miss this episode.

Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Darrel. Welcome to the Guild.

Hey Joe. As a longtime listener and big fan, I am honored and a little bit starstruck to be here today. Thank you for having me.

Awesome to finally have you on the show. I've been trying to get you for a few years now, I guess. Before we get into it, is there anything I missed in your bio that you'd like to share with the Guild?

No. I think that you kind of hit the nail on the head. I liked how you alluded to that mix of industry and vendor experience and the perspective because I think that's one of the things that I've enjoyed most about working at Mable is that kind of wider perspective that I've gotten. But I think maybe if it helps my career Arc kind of started off in tech support and I started doing configuration management and a lot of server backend. My first tech job was maintaining servers for an Internet provider when those used to be local and small. I started working through different roles in software development and sort of fell in love with testing because it was this really nice crossroads between, you know, users and developers and the business and I just found it a very interesting place to be. And yeah, and I just kind of stayed here. But I think that also kind of gives me different perspective. I think just because I felt a lot of the different pain that a lot of different roles in software development have kind of experienced over the years and how that's changed. So now I get to look back on it all fondly. But lots of tears, lots of tears were shed over the last 20 years for sure.

So can you give an example of how you got one of those perspectives of where you have now, like where you were before and what made you go, okay, this gave me a fresh perspective on this situation?

Sure, yeah, absolutely. I mean, I think Agile is one of those things or you know, the notion of shifting left. I worked on my First Agile team 10 or 15 years ago or something like that in the early days and I guess I've just been privileged to work in very forward progressive kind of companies since then. And now working for a vendor, I'm working with a lot of more larger enterprise organizations, you know, financial services and other industries and they're just getting on board with automated deployments and a build system was the first thing I ever automated that was in 1999 or something like that. So it's interesting to see, you know, just how long that journey to Agile and journey to DevOps and journey into continuous testing, you know, are really in various disparate states depending on the organization that you're in. And it's been sort of really lovely I think, to sort of take my experience and kind of lend it to other teams who are starting off on that journey and hopefully help.

Absolutely. I know for myself being older, the good thing about being old is you do have this perspective and you kind of see cycles where before it used to be vendor based tools were all there were and there was open source tools. And now in my perspective it feels like it's swinging back to vendors for certain functionality along with open source tools. Of course the bad about getting old as you're getting old. So other than that, so do you see the same thing? Do you see more? This seems to be in the community like a, not a disdain but like, oh, that's a vendor tool, like almost like it's a bad thing. And from where I've been, like vendors like hp, Mercury, they set the stage for all these new technologies. People come up with these open source solutions that they pretend are revolutionary. And it's like I had this in Windrunner, you know, in 1999. Well, you added it open source, that's cool. But I mean, I don't know where I'm going with this question. I guess the question is.

Do you see people embracing more vendor based tools than they were maybe five years ago, years ago?

I do I mean, you know, there is certainly an element of everything old is new again. Selenium ID is, you know, risen from the ashes. I think that, yeah, I think people are swinging back more towards it. And I'm a, you know, big champion of open source myself and have had the great pleasure of working with folks in different standards bodies. You know, I had friends who did a lot of work with the IETF and I have a very deep appreciation for that sort of work in standards and open sourcing. But I think one of the problems with open source that I've felt myself, there's not really great support when you're in a pinch and you kind of have to solve things yourself or, you know, you're stack overflowing everything or putting something on a GitHub issues list and hoping someone will respond. Whereas I think the benefit of a vendor tool is that you're paying someone money and that, you know, essentially means you have someone on the hook to help you and a convenient place to point your finger when it, you know, comes time to play the blame game. And I think that's just a little harder to do without having that vendor support or even without having a really robust organization, organization of peers who can sort of help you nurture, you know, keep your framework together and whatnot. But most assuredly, every vendor is using some component of open source tooling in their solutions and, you know, we certainly are and I think there's a really healthy respect from coming from vendors towards the open source community. But yeah, I just think there's so much work to do and I like to think that, you know, the more we all work together, the more ideas come to fruition and I'll kind of riff on that and make better tools, better things for us to do our jobs with.

I think you've mentioned this before in some of your other sessions is that every team's different. You know, some teams may embrace open source and they, even if there's not people committing to a certain project, they're technically aware enough, they can code, they can get around it really easily. That's awesome. There's other companies though, that they're not there and based on the culture, they'll never be there. Maybe a vendor based tool could help them there as well. So it's not one against the other. It all depends on the situation. And you could have both in the same enterprise solution as well. So I know Mabel, you folks do a survey and in your survey you talk the state of DevOps. Are there any insights that bubble up there that maybe people may be struggling with tools that you've noticed. Like, maybe they do have tools in house, but maybe it's not meeting a need or any common issues you see when you survey your audience.

Definitely. I think one of my favorite stats to pull off of that slide that I do show frequently is roughly about half of the respondents feel that their approach to testing fits their architecture. And that to me is just, wow, that just sounds very risky. And you know, what are we doing? How much am meaningfulness is coming from these testing efforts if we're not even confident that we can test correctly? I think there was a majority of people feel that their solutions are a bit outdated. I think there are a majority of people who responded who said that they typically spend more time maintaining their testing efforts than they do embarking upon new adventures, which is a challenge. So, yeah, I think the big takeaway is that we still have problems to solve in this realm and there's still more work to do.

Absolutely. So in your perspective, your thoughts, how do you know when it's a tooling issue and not a culture issue? Because I'm thinking I've worked for a large company, very dysfunctional, and every two to three years I'll say, oh, we'll go from UFT to Selenium and that'll solve everything. And then after two years I go, oh, we're moving away from Selenium, we're going back to a vendor tool. And it was back and forth, back and forth. How do you know that your issue is really a tooling issue as instead of say a culture issue?

Yeah, well, I'd say that back and forth probably hints at a culture issue because nothing ever really settles. And clearly there's not good buy in or good confidence in the choices that were made. And so you're like, well, we have to choose something else. Some line from Sondheim show into the woods that was something along the lines of not making a choice is a decision. And I think sometimes people just feel that impulse to change, but maybe the direction that they go in isn't necessarily the best or is potentially misguided, but a good sense of when it's a tooling solution is if your team generally works well. But every time you try to do something, something, you know, where you have to use the computer, things just get hard. The friction isn't with the interaction or the collaboration. The friction is really with the sort of work, the real crux of the work. So, for example, you know, if you're having a bunch of flaky tests all around. Could it be that your code is just bad for your automation? Maybe? Or more likely, is that the solution that you're using? If it's flaking out this much and you know, you're assuming that smart people are using it, it's gotta be something related with the tooling, right? Or otherwise it just shouldn't be that. And you know, that's a problem in and of itself. So I think you just kind of have to feel it. I think it's hard to quantify, but I think certainly some of the things you can quantify are, you know, how much effort are you putting into maintenance, how much effort are you putting into new test creation, and what does that ratio look like? Because I think where most teams lose time is more in the maintenance part of things. And I think any sort of improvements or efficiencies that you can get with test maintenance just, you know, pays themselves off exponentially over time. Because of course, we all want to test more things, but everything we create we have to maintain. And I think that's where we sometimes get lost.

Like I said, I've watched some of your other sessions. You've talked about some warning signs. So it sounds like flakiness could be a warning sign, then maybe time to change to a different tool. Maintenance. If you have a large amount of maintenance, that may be another warning sign. Are there any other warning signs you think there would be if someone says, or is it a culture, a tool issue? Here are some warning signs I know probably is most likely a tooling issue.

Well, one thing that I think could be an interesting combination of both is when testing is lagging behind development. You know, for a while I think it was almost accepted that it's okay for testing to happen. One sprint behind the development, and it's like, well, okay, but are we shipping that?

And most of the time, depending on how agile you want to be, sometimes that's a yes. And I think that is where your tooling maybe isn't making you able to test fast enough or to be productive in the amount of time that your development process requires. And that could be a divide between, you know, well, maybe we're just doing our sprints wrong versus okay, well, maybe we also need a tool like, I tend not to think anything is black and white. Everything is a shade of gray, especially in this realm. But I think some other things you can look at too, that might point to a culture problem is when you raise a concern that you feel that maybe there's not an adequate amount of Testing, and you get pushback. And if that pushback is, you know, along the lines of it doesn't matter, you know, which could be interpreted from a number of different ways, then I think that's a serious culture problem. If I could keep rambling on with an anecdote. I worked with a friend of mine, and she went back to work for her kind of her family business, which is a software company, and she's a product owner. And her sort of stake in the ground was, dad, we don't do testing. And I believe in testing. We don't need to do that. Our customers find their issues just a waste of time. You know, we're. We're always going to ship bugs. There are always going to be bugs. And yes, but maybe we could do this better. And so they did, and they just, you know, happened to use Mabel. But they really started from ground zero and sort of developed a discipline around testing. And I think the discipline of testing is often more important than your tooling choices and whatnot, because, you know, where there's a will, there's a way. And they really created a way will to test their software better. And since then, it's been about a year. The person who started, you know, didn't really know anything about testing, has totally leveled up and is just dynamite. And product is better. Everyone feels better about the software they're releasing. People's morale has improved because they feel like they're putting out a better product. And I don't know about you, but I just love feeling good about the work I do. And so I think a shift in approach or bringing on a new tool can help nurture that kind of change in a company. I think it's time well spent. And, you know, and it's hard, like you alluded to, you know, people switch things all the time because they can't just find the right thing. You have to go in knowing what your requirements are versus your nice to haves and being really open and honest about those, because there's no silver bullet. And you're going to have to make a compromise somewhere. But I think doing that, you know, collaboratively is really helpful. That's a lot of words.

Joe.

Went to my next question about requirements. Like, if someone says, oh, wow, this sounds like I probably should look into the tools, how do they know what requirements they are? Because a lot of times I think, once again, another ramble. People look to Google or they look to Facebook and they think that's the type of testing they need. And their copy could be totally different, totally different risk, totally different needs. So how do they know what to look for then when they're looking for a new tool?

I think we should all start with our users and customers, right? And kind of work backwards from there. You know, what are, if you've identified that there's a problem, you know, in the first place, right? It's like, well, what are users telling you? Where are the problems? Is there a particular area of the application at the component level or is it UI or UX that, you know, we just keep stumbling on? You know, look for those kinds of signs and is there something specific that you can do to address those? So I think look for the obvious problems, you know, some people call those low hanging fruit. I think that's a great place to start and then go more theoretical from there in terms of, you know, how would you know, where do you want to expand coverage and whatnot. But keep it simple. Start with the basics and yeah, show some value really quickly. I always say that one of the first things you can do if you have a CICD system, you know, working in your company somewhere, like get one test wired up to it, it's instant value. And once you have that first one hooked up, becomes more trivial to add more and add more and add more to it because the stage has been set. So I think that's one thing. I think I veered very far from that one. What was this?

No, that was perfect.

Okay.

Yeah, yeah. So it kind of leads into my next question. That's good segue again that you said run a test through the full life cycle to your pipeline and see how it works with your cicd. You do work a vendor, but you also have other experience before working for a vendor. Did you know anything that folks can do to be more successful when they do a poc, say, with a vendor? Are they things you see people that are successful with the POC that they did like these two or three steps that said, well, I wish more companies did it this way, then they would have no, right off the bat, this isn't going to work for me or it is going to work.

Yeah, they come in with goals. The ones who are successful, you know, they come in with very specific goals if they have particular test scenarios they want to test. And you know, let's just say hypothetically one of them is just a no go, like, oh, we just, we really can't do that. We're always very upfront and honest with them about that. You know, what our limitations are. And then we help them identify Other use cases. So I think they come in the successful ones with an idea of what success looks like to them. One of the things that we'll either provide and we're seeing more and more people coming to us with them are scorecards or something like a scorecard where they'll have. Our problem is not enough people can participate in creating test. Okay, well maybe that means that the tools aren't easy enough to use. So ease of use becomes a requirement and so on and so forth. You know, maybe you need to test on five different browser versions while some other team is like, no, no, no, no, no latest browser version. We care about security. We don't, we don't even, we don't want to acknowledge that people are running five year old browsers. So you know, they, they come with these requirements and I had a conversation with someone yesterday where they said, yeah, we have this requirement to test on N5 browse, but that's a lot of testing. And you know, it's like, yeah, it is. It's kind of, I think it's an antiquated requirement that takes its own convincing. Right. So I think knowing what you're looking for, being very open, you know, again, if you're working with a vendor, being open with the sales team and whoever you're working with there, I think everyone now has this weird inherent and mistrust for salespeople. I don't know why, but I've been there too, right? This is my first time working on a sales team ever. But I think one thing to know is that everyone's very busy. The salesperson wants to make sure that they're using their time well and that they're not wasting, you know, a potential customer's time. Because we think that when a customer is buying into a vendor tool, they're not just buying the tool, but they're really buying into the whole team. And you know, what is our release cadence? Are we funded? Well, because that's a real concern. So I think people do who come in with the notion of due diligence as well, feel more successful because they leave with a greater sense of confidence knowing that they made a good decision. I think those are the two really top ones. Really just knowing what you're looking for, which is a big part of one of my topics, is kind of figuring out what it is that you're looking for, but then also being able to recognize when you're looking at it. And it sounds so simple in words, but we typically walk through a POC process takes about three Weeks with Mabel. And it's very high touch. We do a lot of training. You know, we really think about this in terms of enablement and whatnot. And I think that's one of the nice things about working with a vendor too, is that because, you know, we're ultimately trying to sell you something, we want to make you successfully using it. And me just loving testing, I just want to help everyone test their, you know, test their software better. So it's a lot of fun in my position because I get to see so many different kinds of application and understand, you know, all these different challenges.

Absolutely. And I think when you see these other challenges as well, people should know what they're looking for. But how about things that they don't know that they should be looking for? You know, a lot of times people heads down and they're just trying to get their test to pass the pass or fail. But lately, the past few years, you know, I think AI and machine learning have become a real thing. And there are all these other use cases now that been opening up. I've talked to a lot of new vendors. What is maybe new functionality or new areas that people should be aware of, that it's not just like a pass or fail that maybe a vendor tool can help them with.

Yeah, I think going beyond pass or fail is so important. Nowadays. We care about performance, we care about all of these other considerations of our application. And so I think one of the travesties of the past is that so much of our automation effort has been geared towards that pass fail status that we've missed all these other opportunities. And so I'm personally excited about the vendors who are working with tools that do things like real time log analysis or, you know, and possibly generating certain tests or monitors or whatever we want to call them against the application in an automated sort of way. I think it's in its infancy right now, just from what I hear from other folks who've evaluated those tools. But I think it's a really exciting prospect. We have these so many tools, like analytics tools on our websites that are monitoring every single thing that our users are doing, yet we don't have all the insight into what our users are doing. It's like we have to bridge this gap between all the monitoring, our testing and how we approach that, I think. So I think that's. That's one thing. I think becoming aware of new modalities in testing sometimes gives rise to opportunity as well. I think of applitools who, you know, made everyone realize that we do need visual testing at some level. Right. And you know, we have that sort of feature built into mabel. I think, you know, both Sauce and Browser Stack have some level of screenshot compatibility comparison as well. And so that's. It's become a necessary tool and folks don't always get why it's. Why it's a good thing. And so that takes a little bit of education too. And I think you have to talk to more people. I think so often testing teams, especially if you're still in a centralized qa Org, sometimes those decisions about tooling get made in a vacuum or siloed in a way. And I know in a platform like Mable or others that generate, you know, a lot of data and a lot of, of insights, we think more people should kind of come into the platform to be able to have access to all of that data. And so the stakeholders and that tool choice I think become wider and so your developers should have input into your testing tools, even though they may not be the direct user. I think product owners and UX folks have some interest too. We actually have a ton of product and UX folks who do UAT tests through Mable because they're so easy to create and straightforward and they don't need a lot of complex features to do that. Those and so it works for them and they're getting more people involved in testing and they're solving problems they didn't realize they have. So I think it's always a challenge to, you know, how do we know what we don't know, that sort of thing. But I think the more steps you take in the right direction, sometimes the path just reveals itself. And if you're, if you're paying attention and listening, I think some things just become obvious if they weren't already. But I think in most cases, you know, our problems are pretty obvious.

So, Darrell, it's been a while since I talked to someone at mabel. We've talked a little bit about mabel, but what have you all been working on there? Any functionality with Mabel that you're excited about, that you think people should know about?

Yeah, tons. I think early on Mable was sort of perceived as the easy to use automation tool, you know, geared towards non technical users. And there's certainly still a lot there for that Persona. But in the last few months we've been releasing a lot of features focused more on the developer workflow. So we have a command line interface now. You can also run your tests locally in a local browser. We have an alpha right now, a headless Runner so you could execute your Mable test locally, headlessly, which is amazing if you're doing development on a local environment, you know, you have your local stuff spun up and you just want to run some tests very quickly on your local before you even commit or, you know, push any code changes. So you can do that with Maple now, which is great. So kind of another option from running in the cloud. I think we just did a really beautiful JIRA server and data center integration for a long time. You could push issues into Jira from Mable, but that's sort of where it ended. And with the JIRA server integration, you can actually view any issues that were spawned from Maple within the Maple interface and sort of track those a bit, a bit better, more finitely within the Mable paradigm. It's really quite nice. And we're doing a lot of additional work on API testing right now, which I love API testing and so I'm. One of my favorite new features for sure. There's a lot going on. I know at the end of last year we had a slide that was something along the lines of 150 features in 150 days. You know, sort of thing. Looking at our changelog is dizzying, but yeah, there's a lot going on. I think much of it is focused on developer workflow. We did some great partnerships with GitHub and BitBucket on their new pipeline product in the last few months, which GitHub Actions and BitBucket pipelines are super cool. If anyone is interested in exploring CI CD for the first time, set it up. It's really easy. It's a YAML file and some code in a repo. You can do it.

Great stuff. Okay, Daryl, before we go, is there one piece of actual advice you can give someone to help them with their automation test tooling efforts? And what's the best way to find or contact you?

My advice is to explore all of the new things coming out. And if you can't do it directly, look at YouTube. Tons of people have videos of these things in action. There's so much going on. The catchphrase, I think it's become my catchphrase now is there's a we're in a golden age of testing. I really believe it. I think there's so many interesting things happening. If you're not involved in the testing community on like the social media level, seek that out. Because I learned so much on a daily basis just from reading what other testers are doing and it's such a lovely community. You know, folks like Joe here, you know part of that. And, and I think, yeah, I mean, I learned about Mabel from your podcast and you know that. So I thank you for my job, Joe. And seriously, I think that there's so much happening out there, it's always hard to know really what's going on. So I say my best advice is amp up your curiosity right now because there's so much out there to find.

Awesome. And what's the best way to contact you? Darrel?

You can find me on Twitter jb as in boy farris or darrelabel.com d a r R E L I have the least common spelling and I apologize.

Thank you Daryl for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com/a306. And while you're there, make sure to click on the Try it for free today link under the exclusive Sponsor section to learn all about Sauce Labs awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.