Every Experience Matters Report with Marcus Merrell

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast and you're in for a special treat. We have Marcus Merrill from Sauce Labs here to join us to talk all about user experience and also automation. If you don't know, Marcus Merrill is the Vice President of Technology Strategies at Source Labs where he uses experiences for more than 20 years and tests to manage strategic technical alliances. Marcus also works with partners to build robust customer centric solutions around test automation, release management and the entire software development lifecycle. And he started using Selenium way back in 2007. He's contributed to the Selenium project and chairs the Selenium Conference organizing committee. So as you can tell, he has a lot of experience. You don't want to miss this episode. We're going to dive into all kinds of automation goodness, so stay tuned and listen up.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit testguild.comSourceLabs and click on the exclusive sponsor section to try it for free today. Check it out.

Hey Marcus, welcome to the Guild.

Thanks, thanks for having me. It's been great to be back.

Awesome. Awesome to have you. All right, Marcus, what really caught my eye was I think lately I saw a report about from Source Labs on every user or every experience matters report that you put out recently and just curious, I've been hearing a lot more, I don't know why, about the user experience. I know it's an important thing, but why all of a sudden is this such a hot topic among testers in the industry?

Well, I feel like to an extent it's a long time coming and you probably agree with that. I mean, I've been, you know, as travel has started to open up and I've been using more apps for traveling and walking around and making reservations and going to theme parks and stuff like that. I've been just kind of mystified by the, some of the lack of care that goes into some of the experience management as I move through software. And I just, I think everyone has this same sense that, my goodness, like, shouldn't this be better than it is right now? And I feel like the impatience of the average user is going down as the proliferation of software is going up and testers are struggling to keep up. So we're trying to figure out how to help, you know, people scale themselves for this.

Absolutely. And that's funny you mentioned customers have become more impatient. I think you actually have a slide on this in this report on consumers are becoming more impatient. I know a lot of people know that it's like a rule of thumb. If it takes more than I think it was two minutes or something, people bounce. But it looks like these numbers are even getting worse. Can you talk a little bit about what kind of numbers you're seeing from what makes consumers maybe impatient that causes them to bounce or leave your application to maybe never use again, or even why it's important to companies to really care about customers being impatient?

Sure, yeah. I mean, our prevailing theory and is validated by this survey is that companies are losing money as a result of having a less than delightful, less than lovable experience. And what we've figured out, you know, as an industry, as a species, pretty much, is that software controls every part of our life. And most companies aren't the only one that do what they try to do. So if you're working with a bank or a retail site or grocery store or anything like that, A, they have an app, and B, they're not the only ones in the space who have an app. So if you don't offer a great experience, they're just going to go to the other one that does, even including things like banking.

So as a tester, though, who looks after this, like, are customers impatient? Okay. It seems like a lot of these things we talk about user experience, maybe it's just because I'm just older. When I think of user experience, I think of user design and I think, oh, it's like, not really testable. It's kind of subjective. But is this something different than user design?

I mean, ultimately, I don't think it is. I mean, as you know from being a tester yourself for nearly 20 years, I think a lot of bugs that we report happened in the requirements gathering in the design phase. It didn't happen at code time. A lot of the things that we advocate for, you know, people to fix, in my opinion, as testers, our primary job is to protect and represent the user's experience to our people in product and engineering. And I think a lot of times testers, you know, we lose sight of this for brief periods of time. We're automating a test suite. We're trying to get sure. Make sure that we have coverage, you know, of all the different important features and stuff like that. And we sort of take our eye off of the fact that there's really probably on average medium to small app, probably only a couple of little paths through the software that really matter to both the customer and to the company's bottom line. And I found myself in my own experience, you may have experience with this similar we're very, very happy about the amount of coverage we have on say we edit the user profile and you can change your avatar and you can change your nickname and you can change your presence, but nobody's watching out and making sure that on every platform, every time you need to go through the SO software, including matters of accessibility, security, performance, everything like that, there's probably one workflow like a cart checkout flow that matters the most to your business, that needs to be the smoothest every single time. And if you're not protecting that a hundred percent, then probably, you know, wasting your time on some of that other stuff.

So that's interesting. So a lot of times I think you speak about this all the time also is a lot of times people have like thousands and thousands of, you know, selenium tests or whatever kind test doesn't have to be selenium and the tests are not trivial things or things that may not be mission critical to an organization or the big money organization or like you said, this flow that is critical. So should that be part of a user story for every sprint, like make sure you test the critical user flow or how do you make it so that the teams know. All right, I have to test my feature, but I don't have to be so short sighted that I miss out on the whole point of what it is I'm creating that helps the whole flow that's going to help the users really have a better experience.

Yeah, I mean I think you said the word risk in there somewhere. If not, I'm going to, I'm going to say it. I mean I think that what I look at in terms of trying to figure out how to approach testing is much more around risk than it is about around coverage. I spent most of my career chasing coverage, chasing numbers and ultimately probably failing my users until around 2012, 2013 when I was working at my previous job. I'm sure you and I, that's around the time you and I met where I started talking about how if you're not protecting that risk, you're also probably not protecting your users. And so we're trying to move the conversation from pure coverage to more around risk and how that manifests itself. So yeah, I think you're right. You're on the right track there. You have as part of a story, part of a design sprint, you identify the most critical areas and then everything else is kind of nice to have. Like if this fails, if this automated test fails, it's not all hands on deck. So do we really need to test it every build, that kind of thing? So that's the kind of thing that we're talking about now.

Nice. And you actually did speak at this year's Automation Guild and your topic was actually stop over testing, which may sound kind of weird for someone that works for a vendor that probably would want more tests running because you probably make more money, but you don't, which I love. And that's not SUS Lab's mission anyway. So why can someone over test UX then? Because now they're like, stop overtesting Now I need to think about UX experience, accessibility testing, all these other types of tests on top of the normal functional tests that I've been focusing in on.

Yeah, I mean, what we're trying to do is figure out how to incorporate those kinds of accessibility security performance tests in with the existing amount of testing that's happening with Selenium. Because if you think about when you run, for example, a Selenium or a Cypress test, it's doing a whole lot of work in terms of loading up your webpage and hitting the backend APIs, hitting those servers. And it's also doing a whole lot of work at throwing what I would call an intent signal. It's throwing intent signals at that UI in order to figure out what you're trying to do. And so there's a whole lot of passive work we can do to say, hey, while you're in there, you already loaded up a URL, why not do a little scan to see whether or not there are some red flag issues around. High contrast, low contrast, text visibility, accessibility IDs, make sure that a screen reader can index it. It doesn't really cost much extra processing time. So can we give you these kinds of signals alongside the Selenium testing that you're already doing? And so I'm not sure I 100% stayed on track with the question. I think this is the kind of thinking that we're doing now around value.

I think you are, because it sounds like what I think this is. Another trend you're definitely on point here is that leveraging your existing test to achieve or to get other functionality that without doing extra work almost it sounds like. So I was going to say along those lines, then how do you built it into Slim already? If someone's using Sauce Labs, is there any functionality you can just click on and automatically you get some extra benefit of without doing any, you know, extra work?

Well, maybe not click on, but you know, as you're writing your Selenium scripts right now, just as of a couple months ago, there's a way to add two lines of code that will bring in Axe Core, for example. We have a partnership with Deque, who's the sort of the originator of Axe Core, and you can just, with a couple of lines of code automatically get a readout on accessibility for a given page that shows up in our user interface after the test is over and says, by the way, there's some red flag issues we figured out. So it's not as easy as a button click at this point. Hopefully at one point it will be. But we're also looking at, we already offer this for pulling in Google Lighthouse metrics around page load performance that will for, you know, that one I believe is a button click. Could be wrong about that. But essentially it tells you this page is now operating a little bit more slowly than it used to. Was there a change to the DOM that's detrimental? Are there a lot of API calls in the background that could be solidified? Are the images suddenly big when they used to be rather small? And so it'll keep track of that along the way of doing the Selenium testing.

Love it. So I guess this adds extra complexity though, when you're analyzing results and looking at reports. When you have this extra functionality, then does Sauce like bubble it up then as like an insight like, hey, this for the past week has been running X amount of speed and now it's slower. So it may impact your customers becoming impatient or things like that.

Yeah, we have some aggregate functions like that that'll roll up, that'll do kind of trend analysis. We also have a tool that we're working on an iteration right now of failure analysis. So I mean, you're familiar, I know you're familiar with the classic story of Hey, 85 tests failed overnight. Now I'm going to start analyzing them to see what failed. So here's the first one, it failed for reason X. The second one failed for. Oh, it also failed for reason X. The third one, oh my goodness, it's the same thing. We're trying to figure out what was the pattern that happened along those 85 things and then group them together and say 60 of these 85 failures are the same root cause. Fix it once. And that way we give you a starting point for being able to go and fix all these things on your own.

I think that's so key because once again, in my experience from automation, some of the hardest things were maintenance or like just looking at the results. And if you did that, you know, automatically, because we run results, we run tests so often for every build. If you don't get it, able to debug it within a certain amount of time, it's too late. And then people start losing interest. But this is great because you can say, all right, it's 100 errors. Yes, but it's if we fix this one thing, all 100 get fixed. I love that. And you said that's part of it right now. Or is that something you're building in.

Part of it that is under improvement right now? Because right now we tell you what the pattern is. What we're working on right now is giving you kind of a natural language way of suggesting a remediation to say it. Not just that this pattern was the same between these number of tests. We're working on an iteration where that'll say we believe that this locator right here needs to be improved and maybe even suggest a way to improve it, that kind of thing.

Pretty cool.

Yeah, pretty cool stuff we're working on. Like to me, I think you've heard me say this before where I really want the energy around artificial intelligence in this industry. I want that artificial intelligence laser to be pointed straight at analyzing the data that we already have, not spending time trying to replace the human from the testing equation. Does that make sense?

Yeah, yeah. I mean, I completely agree. It's a lot of visual tools sometimes do this. It's great because it bubbles up a problem and then. But it needs a person to say, is this really a problem? So it saves time by automating a lot of things for you, but it still requires a human or a person to actually verify it.

Yeah. And the kinds of things that I'm really excited about are a little bit out there and we don't have on offer just yet. But imagine if you have a website that's deployed that talks to, you know, 35 different API endpoints along the way. And imagine if one of those endpoints is repeatedly failing with a 500 error, but your software just internally retries that over and over and over again. Well, that's all happening inside of our cloud. We should be able to analyze that kind of thing and Tell you, you know, there's a backend service that looks like it hasn't been deployed properly. So you know, kind of doing that sort of analysis, like I said, we're not there yet, but that's what we're looking at to the future is not just failure analysis. What are the overall patterns that we're seeing in your testing that need improvement?

Yeah. I always thought this about SUS Labs because you have so many customers, so many tests running, it's almost like ideal for machine learning. Do you do anything rather than doing a survey to a bunch of customers say how do they normally their applications kind of perform? You almost have like per industry you could say, okay, the norm in this industry based on results we've seen is tests usually take this amount of time or whatever. Are you able to do that type of insight as well where if you run a test it could maybe flag up these types of insights to the user?

Yeah. In fact, a couple of years ago we offered something called the continuous testing benchmark. You probably saw that report back in the day.

Yeah.

And it was giving sort of an industry wide look. And we realized that it was probably a little too general. Like we should separate that out by vertical and talk about banking versus E commerce versus you know, health care, that kind of a thing. We're under revision and talking about that kind of thing right now. We do do some internal metrics that we don't necessarily want to publish for a variety of reasons. But there's also, you know, some data sharing issues and you know, privacy, GDPR kind of stuff that would prevent us from being able to do that wholesale. But. But it certainly is something that we look at on the inside and you know, we're really trying to lean in on machine learning in order to give customers what they need in order to improve the testing that they're already doing as opposed to, well, let's make all your manual testing go away because that's never going to happen. We acknowledge that. Right. So that's what we're really trying to focus on now.

Nice. So also in this report I mentioned something about in the last month there was a percentage of folks that had issues. They found a bug using a site from like a government type agency. So the big knock on a lot of people not getting the right user experiences, the revenue goes down. But, but like if someone's working for a government agency, like I don't care what makes people care. Are there rules or are there certain laws that may be coming into play that's going to make like accessibility testing or ux, even more important.

Absolutely. And regulatory compliance is definitely one of them. Government agencies, when people aren't able to accomplish a task that fundamentally by law, they should be able to allow. There was a moderately big incident several years ago with the Affordable Care Act. You may or may not remember where actually Jason Huggins, one of the founders of Sauce Labs, helped the government try to remediate those problems. And what we have found is that I don't remember what the statistic is. I'm trying to find it right now where we have something like 64% of people who need assistive technologies for accessing a website. 64% of them have trouble with it on a daily basis. And of those over 40% can't do what they need to get done on a daily basis with one site or another. And if you think about how many different apps and websites you interact with on a daily basis, if 40% of those failed and you really, really needed those things not to fail, that would be a horrifically frustrating experience.

Absolutely. So I guess the other thing is, like you said, you're not trying to replace manual testers. A lot of people think you can't automate anything to do with accessibility testing. But it already sounds like you have some things in source in the pipeline that can handle some maybe lower hanging fruit. Like you mentioned something about was it visual with functionalities baked in there.

It's using a tool called Axe Core, which is an open source tool and it sort of scans for things like, you know, watching out for low contrast on a low background, you know, sorry, low, dark on dark background, light on light, text on light background, that sort of thing. It also focuses on. So, you know, the most common problem that I've seen in terms of accessibility issues is that if you have a form that says first name, last name, address 1 address 2 zip code, if the, the accessibility IDs aren't done right, the site isn't instrumented for a screen reader to be able to see it, all the user will hear is text field, text field, text field, text field, text field. And they will have no idea where what goes into what form. They'll see button, button, button, button, instead of submit, cancel, that sort of thing. And so what we're offering is we're not saying this is an audit, we're not saying that if you pass green, you're compliant. What we're saying is that I believe Deque's stat is that around 50% of the most common errors found with accessibility testing are able to be discovered through this kind of automation. And so we're now we're just trying to offer that as a. It's not complete, but it's a. Here's the really low hanging flute. Really big red flags.

Nice. I've also seen and heard that if you make, if you start using the accessibility IDs, not only does it help make it more accessible, but it actually helps automation in general. Have you heard that? Is that something like as an extra carrot you say, okay, make it more accessible and on top of it, you make your tests more automatable.

Yeah. The truth is that before the Domino's Pizza lawsuit, which I can cover in a second if you're not familiar with that one, before the Domino's Pizza lawsuit, that was almost the only way I was able to convince certain people. This is before I became came to Sauce Labs. This is back when I was in industry. That was about the only way I was able to convince people to care about accessibility was to say, hey, this will make your automation easier. And so that was pretty effective. But now that the Domino's Pizza has become such a big issue and everyone is now understanding and very empathetic with their users, when they say we're not only required to do this because we basically provide a storefront that needs to have the equivalent of a wheelchair ramp because we're ADA applies to us, they're also saying we have to do this because it's the right thing to do. And we're seeing that universally throughout our customer base.

Nice. So another thing that is in the report is nearly 15% of the world's population has a recognized disability. So it seems a little low. Actually. I think of myself. I don't see myself having a disability, but even reading this report, I have to up the font size. I don't know if that falls into accessibility or thinking about the user, but that's it does.

It does fall into that. And I could say I think the 15% number isn't people with a recognized disability. I think it's people who likely require assistive technology when browsing the web. I think that was the statistic. So I have the same problem, Joe. I have to adjust the font size on my phone. So what I found recently was that I universally adjusted the font size on my iPhone and as a result, a lot of apps, the funniest thing in the world to me was it was a hotel app that told me, here's how many nights you've stayed in the past year and the number since I was traveling a lot in 2019, the number was over a hundred. And so it said between two brackets it said, you've stayed this many nights. And since I had upped the font size, they didn't account for a three digit font size inside those two brackets. So it said one dot, dot, dot. And I, I couldn't believe it. Like nobody had tested through that. And this is the kind of thing that happens all the time. And this is with a group like you and me, we can figure out how to get around this problem by putting on glasses. Imagine if you had no choice but to have to move through the world with this kind of accessibility tool. And it seemed like the software makers, you know, more or less just didn't care. And so we're really trying to get the word out that this is now not only possible, but somewhat simple using open source tooling that you can just apply to the testing you're probably already doing. You'll at least catch a lot of it. You won't catch all of it.

Something interesting you said there, people have no choice. It's almost like you need to have more empathy as a tester. You know, it's so such a soft skill. But just having that would help you become better at testing and automation.

Absolutely. Empathy for users is, to me, it's a critical part of the craft of testing, which is again why I think that the human will never be removed from the testing equation or they should never be removed from the testing equation. Potentially the human can be eliminated from the really tedious, repetitive kinds of testing that just catches functional errors. But empathy for the user to me is the number one rule of any tester who really wants to appreciate this craft, whether they're doing automation or manual. What was I going to say on that? Also, there's a very great partner of ours, Dequeux. I don't know if you've spoken to them. They would be great to have on your podcast at some point. They have something called the Accessibility Lab, which allows people to, as a group, simulate different disabilities and then use software that is not instrumented for them so that you can put certain things around your experience to be able to understand exactly what your users are going through. And then they take you through your own software and make you go through that. And I think that turns the light bulb on for people really quick.

Well, that sounds powerful. You have to give me an introduction because I definitely like so critical that people hear more about and see for sure.

Yeah, you're absolutely right, we should do that. And in fact, you know I'm working on trying to plan a Selenium conference for sometime in October, maybe Chicago again. I would like to have that lab set up.

Yes.

At that conference. I think that would be fantastic.

That would be so cool. Yeah, absolutely. It should have me out there. Yeah. And I can cover it.

Sure will.

So another thing you go over in the report and I think it was the five top issues users have or have little, little tolerance for. And so I thought maybe we'd touch on them really quick and see how maybe Source labs or other tools that people are using for automation might be able to help. The first one is experience interruption. What does that mean? Is that like a pop up that comes up that you can't close or what's an experience interruption?

I have a feeling it's unclear from the text. I believe that pop ups would be a big part of this, but I believe this is probably more about errors that are happening probably when they shouldn't. I just went to try to book a flight two hours ago and the site wouldn't let me do it. Just don't know why. Went back two hours later and it worked. So it's just the experience interruption. If I'm booking a flight, I'm probably going to try again. But if I'm checking social media and it's down, I'm not going to try again. If I'm trying to get a coupon code, there's eight different companies that do coupon codes. I'm just going to bounce right out and go to a different company that does that.

Absolutely. And three of these seem like things that people normally they wouldn't even know how to test. So I'm just curious to know what you think One is long load times, obviously performance test, but from a user front end don't know how people normally do that. Another one is outdated or weak security and the other one is app crashes. Are these things that could be simulated or tested ahead of time or can you use something in Source that helps.

With any of these long load times? So it's going to be different between web and like a native app experience. And this is the interesting thing about performance is that most of the time when testers, career testers like you and me, when we think about performance testing, we generally think about load testing. And that's not what this is about. This is about whether, I mean it could be if your server is under load, maybe things will go slower. But it also could be you've introduced some crazy DOM structure because of some weird front end that you're using that's causing huge load times and it has nothing to do with the server load. So. So that's where I definitely would. Would recommend looking at page load performance using a tool like Google Lighthouse, you know, and that's something that you can keep an eye on. And we offer tracking of that so that we can do a comparative baseline to say over time, this page has gotten slower. You might want to take a look. As a side note, with one of my former jobs, we actually got penalized by Google because they reevaluated and reindexed how much they prioritize page load time in their SEO rankings and my company's, we stock plummeted when we got punished by that. So users don't tolerate that. And almost more importantly for some businesses, Google doesn't tolerate it either.

Absolutely. So do you have anything built in for Core Vitals?

For Google Core Vitals, Not Google Core Vitals, just Lighthouse.

Google Lighthouse.

Yeah, Lighthouse. So it'll tell you things like time to first paint, time to first interactive, that kind of thing. And that'll be in there. And you can, like, like I said, do a comparative baseline over time. What was the second one?

Weak security.

Weak security. That's one of those things where, you know, Sauce doesn't have any tooling around this at the moment. We work with several partners who can offer this kind of thing, and there is some amount of work you can do to integrate something like zapproxy to do scanning using you alongside your Selenium or Sauce testing. But it's definitely a big issue if someone thinks that their interactions are not secure. Feels like this is an area that the industry has generally gotten better about recently. But at the same time, you see how many scammers are out there with text messages and all sorts of things. So, I mean, that's definitely something that we worry about. And the big thing we worry about is that if you have other kinds of errors in your software, that's definitely something. The report showed if you have other kinds of errors in your software, people will go and assume that your software isn't secure.

They'll also assume, wow, that's a killer for sure. Something that's maybe not related, but maybe is. I was just speaking to someone today that told me the latest Chrome version had a bug in it and that they had to manually update 100 machines because Chrome forces itself to update on the machines. If someone's using a service like a crowd, one of the benefits, I think, of using a cloud provider like Sauce would be that it would be easy to fix that, I would think. Is that true? And that would then play into this whole security type bucket.

Yeah, I think we tend to ship browser new browsers within 24, 48 hours, and if something like that would happen, we would immediately shut off the ability to connect to a node that was. That showed a browser like that. So, you know, internally at Sauce, it might cause some headaches, but for the customer, that's the kind of thing that you shouldn't ever have to worry about.

Seamless. Love it. How about app crashes? How do you emulate an app crash?

Well, how do you simulate an app crash? That is, There's a lot of different ways to do it. You know, you can bring the backend server down, you can do targeted fault insertion, you can do whatever. But really the thing for me is that kind of testing is of somewhat limited value just because it's always synthetic, you're never sure if it's real. And the software behaves differently in test than it does in production, as you're well aware. That's one of the acquisitions that we made about a year ago was a company called Backtrace, which does app crash error monitoring, that kind of a thing thing. It was purpose built. So for ad tech and the gaming industry, it's beloved of game developers in that when you turn on a release or turn on a feature flag or something like that, 4,000 people might experience a problem all at once. But it all gets deduplicated and indexed and symbolicated and sent back to the developer with a root cause analysis that says here's how many times it occurred, but it's all the same thing. And really powerful tool for helping to diagnose once something gets into production, how things like that have gone. The value of doing synthetic testing to me is a little bit up in the air. I really, I'm a fan of this new. I think of it kind of as a new idea of releasing small amounts of traffic, say cutting in 2% of traffic into a feature flag and then watching the monitoring in production as users go through the software when it's first deployed. Because my theory is that it's a lot easier for you to catch that thing with this kind of software in place. It's a lot easier for you to catch that kind of thing within a minute than, you know, waiting until someone posts an angry Reddit post. Like, you don't want that to happen.

All right, absolutely. So it's funny you mentioned Backtrace. I'm actually running a conference with someone that's I'm in a partnership with someone that's doing a game development. So it's Game Dev Guild and one of the sponsors they got was backtrace and I had to send an invoice to Sauce. I'm like, what? So I don't know if anyone's like me, like, if you've been to Sauce, I don't know how people haven't been to Sauce lately. It's like you've done way more than what you started with. Source has been a sponsor since 2014. You have things for cross browser testing, mobile app testing, low code testing, error reporting, mobile beta testing, API testing, ui, visual testing, test automation. I think I saw some low code. So what's going on? How do you keep track of all that? How do you know? Like, I guess this is a good indicator of what maybe is going on in industry because obviously you're acquiring these companies, so you must see a need for it. How do you bake it into the pipeline? Then let's people know that, hey, not only do we do, you know, you can run your test against all the devices and all the browsers, but as part of the service you also now do X, Y and Z, Right?

Well, it's challenging because when you acquire four companies in 12 months, just getting a handle on just the personnel issues alone takes up a lot of that time. But then trying to integrate different disparate APIs and different user interfaces, all that stuff, it's just as complicated as you would imagine. But we believe that the vision is very sound and that as long as we don't overwhelm our users with. Now try this. Now try this. Now try this. At every moment of the day. We think that for the moment, we bought these businesses as a fairly, you know, well functioning businesses that could stand on their own while we do the integrations. And now that we're getting a better handle on it, we've been, we're about 12 to 18 months, depending on the integration we're talking about, and we're starting to get a handle on trying to sell what we're calling the DevOps test toolchain, which is to say that every step along the sdlc there's a testing component and that if you go with our brand, you don't have to deal with so much procurement, so many different vetting of vendors. To me it's, you know, as a practitioner, as someone who comes from industry, I'm just sitting here marveling all the time at how much I would have used this platform when I was Doing the job now it's just a matter same here.

I've been unemployed for like two years. So like when I see this, I'm like, oh my gosh, if I had this at my enterprise company back in the day, I mean the headaches. And I'm not just saying it because you're a sponsor. I mean we actually use Sauce Labs at this company as well. All this newer functionality would have just blew my mind. I guess one of it. This might get off on a tangent here. Not necessarily UX related. It is. You required something. I forgot I could never say the name Atomic Q or something. Atomica. I don't know whatever it is, but it was a low code, no code solution. You know, you're a real automation engineer. You've contributed Selenium. What are your thoughts on these low code, no code solutions? Because almost every company that hits me up nowadays is like a no code, low code solution.

Right, right. So my take is, I guess you would say fairly nuanced and I'll try to be succinct here. First Autonomic. The reason I love the product is because it gets us into, well, I guess you'd call walled garden testing. So it opens up to Salesforce Workday Oracle SAP the ability to reach inside of. So we had a lot of customers for years say how do I write Selenium against Salesforce? And we'd say under no circumstances should you write Selenium to automate Salesforce testing. It's just the product changes too often and you're setting yourself up for failure. So a lot of customers would say we've got QA teams that are being approached by the business side saying they need, need help with automating Salesforce workflow changes. What do we do? And so for me it represented a beautiful opportunity to get into that space and actually have an answer. When they say we need to be able to automate Salesforce. Here's how you automate Salesforce and the beauty of it is, as opposed to a company that I probably shouldn't mention that is dedicated solely to doing Salesforce automation is that we integrate. A lot of companies are building integrations with Salesforce into their own products and with this product you can test all of that workflow as well as the Salesforce workflow. So as to the broader question about low code testing in general, I'm a believer in the future, as a practitioner, I have had not as much success. And so, you know, I've got, I'm not necessarily a believer and I, I gotta Be careful here because I, I'm friends with a lot of people who work for these vendors and I respect them greatly for what they're trying to do do. But it feels to me like the attention that we're spending on trying to say, hey, we'll automate all your workflows would be better, saying hey, look, we're going to assist you in these fundamental areas here and help you scale that testing. But the promise of replacing the human, which I see a lot in the marketing materials and whispers, it's just to me it's the wrong focus and it's selling snake oil that I think shouldn't be sold as having been someone on the selling end of it for a long time. It doesn't work the way people say it does.

Absolutely. So interesting thing you mentioned there. I think a lot of these solutions are better for package based applications. And what I heard there's a lot of pockets of in enterprises where these applications are not only tested but they're usually mission critical. And so to not be able to automate it at all using like a low code solution that was built for like a package based application has a lot of benefits. So I think, I don't know if you see this as a tester. A lot of times people get fixated on one tool for the whole enterprise. Like we use Selenium for all our testing and therefore developers need to use it. You know, everyone needs to use it, even business users where as maybe if you're nestat, you're a tester, you use Selenium. If you're a developer, use Cypress. If you're a business user and you're using a package application, use this low code, no code solution. Is there a problem with that? Do you see that as being an issue where maybe it's okay you have different users within your enterprise that may benefit from one tool over another as long as you, you get the job done, you're creating a better user experience for the user. Why would you care?

Yeah, absolutely. So our CEO Alad Miles, I don't know if you've met him, he's great, he's brilliant. He has a thing he says all the time called the tyranny of or. Let's not be caught up in the tyranny of or. For us this is an end. We incorporated Cypress testing into our platform because there's a legitimate need for tool like that that's developer friendly, that's you know, a little bit lighter weight than Selenium, doesn't do the full black box. That test that Selenium does, each has their place. That's why we also support puppeteer playwright Test Cafe on our platform. That's why we support separate API testing. We should have had that feature years ago. We believe that it is all an and and that for many, many years, you know, Sauce Labs was purely Selenium focused. And people who generally use Selenium, you're very familiar with this and this is a Selenium like board member telling you this, that people have overused Selenium for years. And as much as I believe it absolutely has its place at Sauce, what we're doing is pulling the camera lens back and saying Selenium occupies exactly as much space in the process. We just believe that it all ought to be, all of the testing should be all on the same roof and there's not just one size that fits all.

Absolutely. I keep saying it. Simon Stewart, the creator of Selenium webdriver, says it himself. He says if you have More than like 3,000 review using 10,000 selenium tests, you're not using Selenium correctly. You're not testing correctly. So it's not just someone from a vendor. Marcus is very non vendor agnostic anyway, but I totally agree with you. Just a great point to bring up for sure.

And that's why I'm glad, I mean, I was so excited when we bought API Fortress, because in my opinion, if you have 30 selenium tests that are sort of submitting the same form over and over and making sure that the results are the same, like 29 of those should be API tests. And so now we will help you. And it's something that, that I've come to sort of a profound realization recently, which is that now that we have this tool in our tool bag, we can start to be really, really prescriptive and honest and direct with customers about how to approach testing as opposed to someone who only sells Selenium, they're going to just sell you more Selenium. So it's, it's a great pivot for us.

Love it. And it's right on point. What I've been seeing, I speak to a lot of people, so I love seeing this. And that's one of the reasons why I love having Sauce Labs as the official sponsor of the Test Guild Automation podcast, I guess. Marcus, before we go, is there one piece of actual advice you can give to someone to help them with their automation testing efforts? And then what's the best way to find contact you or learn more about Sauce Labs?

Well, on Twitter I'm M E R R E L L. We could probably write that down somewhere. LinkedIn I'm just like as it spells like just on LinkedIn. I don't do much tweeting, mostly retweet, mostly some social justice, racial accessibility kinds of issues, stuff like that. But for the first question, I think that if I were to give my early career self piece of advice, it really would be it's two things that are kind of representing the same thing, which is find out what is the most critical thing that your business depends on. As Titus Fortner says, what does the money want in this equation? So for an E commerce company, it's the checkout flow. If you make sure that your checkout flow is rock solid on every browser, every app, every native app experience and you use journey monitoring tools to verify that correct events are being emitted to your business intelligence processes, your CEO will know your name and know who you are. If you're able to say I was able to save you a lot of money because we were about to ship a broken analytics signal and we didn't ship it or I found the problem with one that we did ship. If you can isolate the highest risk area and use analytics to prove that the testing is correct, it'll change your career, change your life.

Thanks again for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com a396. And while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show, and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full Stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.