What is Quality Engineering with Dan Belcher

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. And today we have back on the show Dan Belcher to talk all about QA and quality engineering and probably all other automation topics. If you don't know, Dan is one of the co founders at Mable, which is a really cool AI automation solution that helps you for all your DevOps types of needs. And prior to Mabel, Dan has worked a lot of different great companies like Google, Microsoft, VMware and he brings his awesome experience to the startup world. So he has a lot, a lot of experience in this space with technology. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment. For every framework, browser, OS, mobile device and API, get a free trial. Visit testguild.comsourcelabs and click on the exclusive sponsor section to try it for free today. Check it out.

Hey Dan, welcome back to the Guild.

Oh, thank you very much Joe. It's great to be back.

Awesome to have you. As I mentioned, you were on the show I believe over two years ago and just curious to know, in two years? You know Mabel, when I interviewed, it was before you even went live, I think, with the product. And I'm just curious to know, was there any pivot between what happened when you first launched the product, what your thoughts were, what it was going to be and what's involved to now?

Yeah, when we started the company, we were really focused on browser testing. We heard from a lot of people that that was really hard, especially if you didn't have programming background. And so yeah, we spent those first few years completely focused on the browsers. As you know, those are. It's a really difficult challenge to do that with low code. And the thing that's changed over the last couple of years is we've broadened that view to include API testing, insight around performance, and you know, we recently launched accessibility checks to be built into those end to end tests.

Absolutely. And I love that you're right on trend. Accessibility testing the past year has been hot, hot, hot. First question is why? And the second question is what made you say, okay, we definitely need to bake this in now to our solution?

Yeah, I think Joe, my investors wouldn't like it. But the first reason why is I feel like we're doing the right thing for the world. Right. When I'm working with software teams, accessibility is. It used to be something that you'd say, well, what percent of our users require improved accessibility? You do some business math and so forth. And now it's an inclusion and equity issue. Right. And so our customers, those teams, they want to build accessible applications because it's the right thing to do. And frankly, I'm just honored to help support them in that endeavor. The other thing is it makes sense, right? Like we had a lot of customers tell us that they've built out all their end to end testing in Mable and now they have to go recreate those end to end tests in a different tool just so that they could run accessibility checks within them. And so we said, well, why don't we just give you that ability to do it inside of Mable so you're not duplicating your efforts.

Nice, nice. And I misspoke. It's been more than two years. I think when I first spoke to you was in 2017. So it was a while ago. So it's great to see that Mabl not only was in its infancy now, but it's still thriving. It's thriving now. So really cool to see as a startup how that works. So how do you know what features to work on then? Obviously accessibility is going to be something that you want to add in, but when you were a startup, how do you know going forward, like what things make sense to build in since you were so small and now obviously you've grown so much.

Yeah, there's no silver bullet. I've seen that now across a bunch of startups. There's no one source that you can go to for the answer. Of course it starts with customers. Right. So we have a customer advisory board. I think it's really important to listen closely to understand what do those customers really need. And sometimes they won't tell you what they need, they'll tell you what they want. And you got to kind of read between the lines to get what they need. The other thing, fortunately for us, is I've seen a lot of customers kind of hacking what we gave them to achieve some other goal. Right. So for example, API testing, we have these API steps that you can run inside of a browser test, right. Maybe you're fetching some data or something like that. And then all of a sudden I saw customers creating end to end browser tests that all they did was make API calls. And I said, well, you probably want API testing Right. And so when I started to see that type of behavior that I said, you know, this is a good signal that people see it as a natural extension to our product to add these other capabilities. Another one is performance. Right. We find customers using us now for load testing, stress testing. We do not have a load testing and stress testing product in market today, but they sure are giving us a lot of signal that they'd like to see one.

You know, once again, another trend I've been seeing is leveraging your existing tests to do other of these non functional tests. So it sounds like you've been seeing the same thing.

Yeah, absolutely. The powerful thing for us is because we run in the cloud, we run on Google's cloud primarily, we can observe a lot of information as we're executing these tests and process them really quickly and really efficiently. And that lends itself really well to non functional testing. Because if you're on prem or you're writing scripts, trying to manage the infrastructure to analyze traces, metrics, visual information logs, accessibility, test output, that's a big challenge for you to try to deal with if you're rolling your own stuff. But if you're running in the cloud, there's lots of good services that those cloud providers give you to help build this stuff out.

Yeah. And I assume you'd have a lot of data going on too, that you could find a lot of key insights of what maybe a lot of people are doing wrong. What a lot of companies that do, right, what they seem to be doing more of is that have access to that type of data where you could pull out insights and go, maybe we should add that or maybe we should, you know, work on that.

Yeah, absolutely. A great example of that is, you know, part of auto healing is you have to build these heuristics and have different weights for different attributes and you know, back testing those, any changes to those heuristics to say, would this lead to more reliable tests or less reliable tests? We can actually back test changes to that algorithm offline, right. Against historical test runs and say if we had had this since the beginning, it would have made, you know, 10% fewer tests pass or fail or what have you. So yeah, absolutely. The data helps us drive more intelligence. Nice.

Now, when you first came into the market, you know, AI machine learning, I think you one of the companies, the first companies to really kind of promote that. I think your messaging has changed a little bit more going towards like quality engineering. Am I reading that right?

Yeah, it is. We struggled initially whether we talk about the AI stuff or just do it. Right.

Yeah.

Right. And I think it struck us, you know, if you use Waze, we don't think Waze is the AI driving, you know, GPS tool. Right. Waze helps you solve a problem. It just so happens that there's a lot of intelligence built in. And. And it's also more about the mission, our mission. You know, whether we're using machine learning or a hammer or a drill. Right. We want to help empower people to participate in automation, to ship better software, to do that faster. And we'll do it with whatever techniques we need to. Right. What we've realized is that as an industry, for us to really get better, we have to embrace some of these principles around quality engineering. And so we're doing everything that we can to help people do that.

Absolutely. And I want to get to quality engineering, but it's the same thing with terms like AI, machine learning, low code, no code. It seems to really drive some people the wrong way. And I like your approach how it's, let's just do what we're doing. Who cares what we're using behind the scenes? And if it works, it works. And you've been around for five years now, so obviously the technology is working. So it seems like you're getting more and more customers from what I see as well. So that's awesome.

Yeah, the growth is really fun.

Yeah, yeah, absolutely. So you mentioned, you know, this thing going towards your messaging, quality engineering. You know, when I started, there was a term qa, but quality engineering. I'm not sure. Is QA and quality engineering the same thing or what the differences are between the two?

We don't think so. We think there's some pretty significant differences in. When you're looking at a team that's really embraced quality engineering versus traditional qa. The first is that the quality engineering kind of focus team is building quality from the minute a developer is working on a branch locally and all the way through to production. Whereas sometimes the traditional QA team is sort of receiving a deliverable from the development team that's ready to be tested. Right. The second is that the quality engineering focused team is thinking about coverage in a much more ambitious way. Like they want to have enough automated test coverage that they can truly trust the test coverage. And the difference between feeling like you have good test coverage and great test coverage is the difference between being able to deploy kind of automatically versus needing to do a bunch of manual verification before you feel that level of comfort. And then the last thing is data. So quality engineering focused Teams are, they're measuring, they're changing their processes, measuring again, seeing what's going to improve. And they're also using that same data for identifying issues with non functional requirements, whether that's performance or UX or accessibility or otherwise. So I think those are the three real pillars. Incredible coverage, building quality throughout the entire pipeline and then using data to improve your work and non functional quality, not just functional.

All right, so let's just put it towards real world then. So for coverage, how does a tool like Mable help with coverage then for quality engineering?

So there's a few different ways. So one, Mable gives you kind of at your fingertips access into understanding what's been done related to a release. For example, so I'm in a sprint. I need to know how many of my tests have been exercised, you know, what's the percent, pass rate, fail rate, how many of the tests have been updated, have I added new tests during this release? Right, that's, that's one example of coverage. But we take it much further. So we'll tell. Teams can use Mable, for example, to correlate coverage, test coverage with usage. So I can look at here are the most frequently accessed pages or here are the new pages in my application and do I have effective coverage over those? That's not code coverage, that's coverage versus usage. Does that make sense?

Yeah, absolutely. So it's more like risk based then. Because you may think, oh, we have 1,000 tests on this page, but wait a minute, like we only have 2% of our users are actually using it and adds no business value per se or money generating activity.

You got it. Yeah. And then there's other dimensions of risk that we measure at mabel. One example is how complex is it? We count. We'll say this page has 39 interactable elements. And so I think that warrants more test coverage potentially than a page that only has two interactable elements. And so we in those coverage reports in Mable, we give you the summary based on usage. How complex is the page? How close to your home page is it? There's various ways that you can cut and slice that data. So that's a good example of what we mean with quality engineering and using data. Right. That potentially teams wouldn't really have without us.

Nice. I think when people think about tests, they think about functional tests. But it sounds like with these quality reports, can you say, look, a functional test. Yes, we know this page is complex, but maybe we should tell our developers you might want to have more unit tests because obviously this page is really important. Do you see people using the data like that as well?

Yeah, we think that it's the data that's collected in testing can be used for many different use cases outside of just improving your functional testing. Another example with that is the integration with Maple and jira. So when you create a test fails with a click of a button in Mabel, we create an issue in jira, but we attach a Chrome trace, a screenshot, the HAR file of all the API calls, and the entire DOM snapshot so you can get rid of this back and forth. I can't reproduce that issue that you found in your testing. Right. It's all right there in the issue in jira. So that data we really do see as valuable in a lot of different use cases.

Great. So you also mentioned quality engineering critical for DevOps in the pipeline. So what do you mean by quality engineering in a pipeline? How can tooling like MABEL help then within the pipeline itself?

Yeah, we noticed early when we started off, our notion of the pipeline started with something getting deployed. Right. So you would tell Mabel, hey, we deployed a new branch to this staging environment and then we would run tests automatically based on that. What we learned is actually that's almost too late. Right. What you want is to get the testing, a good amount of testing done before the branch is merged to your main branch. Because that's kind of in a, you know, CICD model. That's when it gets on the train that's destined for production. Right. And so to make sure that you get good testing coverage from an automated perspective before merge, we have to run the tests inside of CI. So we delivered a CI runner that can execute tests headlessly before it's deployed against, you know, deployed to some environment. And then if you're going to have good tests running in CI, you have to have developers validate them before, you know, they put the changes up. Right. And so then we had to create a local runner that, you know, developers can use to run those tests before they even put it up for review. The great thing is we now call that a unified runner. We use exactly the same technology to run tests locally in CI and then in the cloud.

Cool. And you also mentioned for quality engineering, the third pillar was data. So you talked a little bit about how data can help find maybe risk areas of an application that may not have enough coverage. If you're running in the pipeline, is there any other data you can have? I know a lot of times, like your management or you have like a quarterly meeting with your high level Managers and they don't know what's going on with automation. Is this the type of data also that you're talking about?

Well, that's probably the conversation that I'm having most often today with CTOs and heads of digital is as they're going through digital transformation or DevOps transformation, they want to know what those KPIs should be for each of their teams and then to roll up to look at the initiative overall. And so the thing that we've done, there's a lot of data that's curated inside of Mable, but we also can stream that data to BigQuery, Google's kind of data warehousing solution. And a lot of people are using that to connect to custom dashboards that are tailored to their team's requirements.

So with Agile and DevOps, is everyone a quality engineer or is that a role or is that just a movement almost?

I think it's a set of practices and knowledge for sure. I don't think everyone is a quality engineer. I think we'll end up in a place where there's training and knowledge that you want to develop. To really call yourself a quality engineer, I am different than a lot of the leaders in this space and that I don't think that you can just tell the developer when you wake up tomorrow, you're going to be a world class tester.

I think there will be people who specialize in quality engineering. Likely they come from a testing background and they've developed additional skills that help them along these other dimensions, thinking more holistically about the pipeline and using data to improve. But yeah, I think there's definitely a place for quality engineering specialists that are supporting the rest of the team and kind of adopting these types of practices now.

It's interesting you would say that. Just the reason why whenever I talk about like an AI machine learning technology like Mable, I get people saying, oh, I'm a tester, I'm gonna be out of a job, there's no need for me anymore. It sounds like almost the opposite. And you talk to a lot of customers, so I'm sure maybe your views also shaped on what you're seeing in the wild as well.

Yeah, I've been in this space for 20 years, Joe, and I feel like the better analogy is that we've all been so far underwater just trying to keep pace with functional quality as the pace of kind of development velocity has gone up, that all we're trying to do is use the automation, the core kind of test automation, to get everybody afloat and then they can do all the things that they've always wanted to do. Right. Focus on ux, focus on getting ahead, focus on strategy, focus on collaboration, focus on non functional quality. Right. Making sure that we're accessible. These are things that we really want to empower testers to finally get to spend time on instead of the reactive stuff that, you know, so many teams have been kind of stuck in due to the fact that it's just been so hard to get core coverage around. Functional.

Absolutely. So we talked about accessibility testing, how you recently added. Are there any other trends or any other things you see on the horizon that you think maybe testers or developers need to be aware of or technology that you think may be gaining steam in the next few years that maybe they need to get ahead of?

Yeah, for sure. If in the enterprise we still see teams that are in the early stages of their DevOps and continuous delivery journey. And so if you're not thinking about automated pipelines, I think that this is a good time to get focused on that. As the bar for UX goes up, I think we're going to be talking a lot more about how do we validate that we're delivering great user experiences all over the world. Right. Because digital transformation has led a lot of companies to the point where they can access markets and customers in new places. And so there's a lot of new kind of global concerns. Right. So how do I validate that the user in Spain's getting the same great experience as the user in the US as the user in Japan, I think is an area that's going to be more interesting. Accessibility we've talked about, and then load and performance I think actually gets more interesting too because you're relying on lots of third party components. We have big distributed systems. We got to make sure that we can validate that they're working correctly at, you know, under stress. So those are a few that I think come to mind.

So when I think of ux, I just think of design. So when you say ux, what does that mean? Like how does it performance, how does it look or how localization, if it translates correctly, what kind of issues do you see in that area?

I see two angles to it right away. So one is thinking about things like responsive, right? Yeah. Sometimes it really does take a human view to understand whether this thing is correct from a user experience perspective. When I'm looking at it from a mobile viewport or tablet viewport or sometimes even large screens. Your app wasn't designed to scale to some of these incredibly high resolution screens now. So I think that's one part of ux. The second part of UX that I'm more excited about is what we were talking about in terms of empowering teams to kind of uplevel their impact. So what I want is my testers to be coming to the table to say our automated tests know that this thing functions. So I'm going to give you some insights into where the UX is wonky. Right. It's inconsistent in this new feature compared to the patterns that we've established. Or this part of the user flow just doesn't make sense. Right. And so testing for, you know, is it a good user experience? I think is a good area of growth for us as well.

Would it have a score or something like, hey, UX experiences, blah blah, blah. I don't know if it's that much of a science yet or is it just like an insight like, hey, this past few runs, this looks a little wonky. You may want to look at it. Human user.

Yeah, there are a few different ways that we're looking at scoring. Now the good news is Google is giving us kind of common approaches to looking at, for example, performance. With the new web vitals stuff, we have accessibility scoring. Now the one that I think people will look at from our perspective is risk scoring. So we want to know when there's a new deployment, how risky is it, how much change is embodied in that, how much of the user experience is affected? Right. I think that is something that we can score that risk.

Ooh, that's interesting because a lot of teams, a lot of times they have thousands of tests, new release, run a thousand tests, some fail. Is it because of the change? I don't know. Should we even be running this test? I don't know. So Maple has a way of tracking, does it know like code that's checked in what areas have been affected and then maybe map what tests may be able to run against that change. Is it that that advanced yet done?

Well, Mable really tries to take the view of the user. So what we tell you is here are the things that visually change to the user that we observed in our test that don't normally change. Right. And so you could say, well, okay, I deployed this to staging and Mabel told me that there are 500 pages that change that don't normally change. And so I don't want to deploy that automatically. I'd like to get some manual testing to make sure that that's appropriate. Mabel will also tell you that performance from an end user perspective, the perceived page load for that release that just went into staging is much slower than it was with the previous release. So again, that gives you a sense of risk. You know, do we have a performance issue that's part of this that we need to address before we ship?

So it sounds like you're getting a lot of insights, a lot of data, but, you know, teams are struggling to ship their features really quick in a two week sprint. Even quicker, two hours. I've seen teams ship software. So does it rank? Maybe what's considered high or critical, A medium or something so people know, oh my gosh. All right, this is definitely high. This is a nice to have, but maybe we'll put it on the backlog or something.

Yeah, you definitely need, I think in two dimensions. So you need the things that are going to fail the build, right? And then you need the things, you need the data to understand how you're trending. And so our approach is, let's take accessibility testing. You can choose to fail your build or fail your tests if there are critical accessibility issues, but you still want to log all of the accessibility issues. And so you could say, I want to move quick, don't fail my build unless I have critical accessibility issues. But we still want to look at the trend over time across all levels of issues to know whether we're improving or getting worse. Same from a performance perspective. I may not want to stop the pipeline if performance gets a little worse, but as you know, performance can get a little worse and a little worse and a little worse and all of a sudden you're in some debt. That takes a long time to pay off. So we work really hard to make sure that teams have both options. They have the trend data and they have the ability to set rules to say we want to fail if this criteria is met.

So if someone's using Mabel, all this features and functionality you're talking about, is it baked in already or do they have to add in like, oh, we have accessibility now. So now we need to change all our tests to add this code snippet or something to. Or is it just flicking a switch and it's all accessibility now?

It's a little of both. So performance is something we measure automatically. Accessibility, there's no, you don't have to add any code. That's the great thing about Mable. But accessibility, you can take an existing test and you can say at any point, I want you to. Mabel, I want you to run an accessibility test at this step. At this step. And Then another one at this step, another one at this step. So there's definitely some things that come built in and then you can sort of build on that by adding more checks to your test.

Very nice. So have you seen any pushback from companies saying, well, quality engineering is not really important to us, or is it? If you use the right tooling, it kind of causes people to create quality software. Is that possible?

Oh yeah. The tools can't do the job on their own, that's for sure. As you know.

Well. Oh yeah.

I think one thing we culture is terrible. Yeah. One thing we do see is customers sometimes our teams early in their life cycle. You know, we have this drive as to empower developers to do as much as possible. Whether that's security, testing, monitoring, you know, writing the code. Let's do some ux. Right. You can. We have this notion of these superhero developers sometimes and especially with relatively kind of younger teams that haven't been around that long. And I think, you know, our view is, and what we see is when you get to big complex apps with large, you know, sets of tasks that, you know, they need to view the testing as a product unto itself and that that requires a lot of expertise and you know, care and feeding and strategy and so forth that sometimes developers don't want to spend their, you know, their time on. So you know, that's the one thing I think we get sometimes this idea of kind of trivializing the effort of testers and quality engine and quality engineers and you know, we want to make sure that as teams scale that they're able to, to really invest in, invest in quality and that it's not just one role's job to do that?

Absolutely. Okay, Dan, before we go, is there one piece of actual advice you can give to someone to help them with their quality engineering efforts and what's the best way to find contact you or learn more about mabel?

I think it's really important whether you're starting a new QE initiative or looking to up level your existing efforts to develop a vision. Right. To start with where do we want to go as a team and to make sure that's a long term view that you get buy in from all the stakeholders in your team and you really want leadership, executive level support in that vision. Because one thing I've seen is these initiatives, they sound great at the start. You have one good champion and she's driving things to get going and then maybe she leaves or maybe the winds change direction internally and things get blown off course. And so I think you need to get that sort of alignment up and down the organization to say we're going to go do this together. And at Mable, we believe that it's really important to partner with our customers on that. So customers, the thing that they love most about us is our support, whether that's customer tech support or sales engineering or sales team or customer success technical account managers. And so we like to work with customers on that vision and roadmap for them. And so I'd say the tactical thing you could do is try it out right? Can sign up for a free trial mabel.com two week free trial engage with the team on what are your goals and how can we help you sort of craft that roadmap and vision for up leveling your quality engineering efforts internally? Oh yeah, I'd be happy to chat with anybody. You could send me a direct message on Twitter. It's Belcherdan B E L C H E R D A N thanks again.

For your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a398 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to test guild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.