262: Testing Torchbearer of Truth with Mush Honda

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe and welcome to the show. Today we're going to be talking all about quality and being a torch bearer of truth about quality to our teams as we speak with Mush Honda. Mush shares why we've over the years have combined this need for speed with our DevOps or continuous testing, combined with the need for the mindset of testers of not being the keeper of quality, but rather more of a torchbearer to be able to make sure our teams are delivering on the confidence factor of our software when it gets into the hands of our customers. And that's what we'll be talking all about today. Check it out.

Test Talks is sponsored by the fantastic folks at Sauce Labs, the cloud based automated testing platform that eliminates the need to maintain your own selenium grid and test infrastructure. Try it for free today. Visit testalks.com and click on the sign up now link under the homepage sponsor section.

Hey, Mosh. Welcome to Test Talks.

Thank you.

How are you? Awesome. It's great to have you finally on the show. We've known each other for a while. We were on a webinar maybe a year or so back and you have a lot. So today I thought we'd talk about your insights all around automation testing, but also around continuous testing and delivery. But before we get into it, can you just tell us a little bit more about yourself?

Sure, absolutely. So I've been in the testing space for close to 20 years now. I started as a developer, went to school for programming and realized that it was a lot more fun to critique out and find issues in code, especially if it's not my own. But that's essentially where I realized I enjoy testing testing more. And you know, over the years I've had the honor of knowing a lot of the other thought leaders in the space and just learning and applying skills and sharing with other team members. I've been doing this, like I said, For 20 years now and really has become a passion. So can't really say it's worked for me. So it's been a great journey.

Very cool. Is your title VP of Testing?

It's VP of Testing Services. I don't know why people always leave out the services aspect.

I was going to ask what is a VP of Testing? But a VP of Testing Services makes a lot more sense.

Very popular tester I would say.

Very cool. I guess one of the first things is as I was researching for the interview, I know you did a session recently on continuous testing, continuous delivery, and I thought we'd dive a little bit into that. And so the first part of it would be as we're going through these digital transformations, as you mentioned, you've been around for 20 years. What have you seen in testing that has changed most recently? That maybe hard held beliefs about testing that people need to get their minds around that have changed in the last few years? Maybe.

So the first thing I would say is people thought testing was easy and obviously it's not. But I think for the last, I would say close to maybe five years or so, I have seen this tremendous.

Focus on speed.

And sort of getting into the mode of going out as soon as possible, beating the other guy. Right. And from a business perspective, that absolutely makes sense. The key change that I have seen encounter, especially by testers, is this change in the mindset of saying, hey, I don't own quality. I am more of a light bearer, a torch bearer, of being able to explain the confidence that your product going out the door would have when it gets into the hands of the consumers or its intended audience. So I think what I've seen over the last five years is combining this need for speed, for lack of a better phrase, combining that with this mindset of I am not the keeper of quality, I am somebody who needs to make sure I am able to deliver on the confidence factor of software. I think that combination has finally sunk in for the testers. And what that has resulted in is also a shift in the mindset with how people look at automation. So if you recall when automation tools were becoming popular and were gaining momentum, most of the traditional testers who validated systems or looked at the capabilities of system and evaluated that they were feeling threatened, saying that, oh, automation is now going to replace testers. And I think especially over the course of the last five years, people have definitely realized that automation, automation is more of a tool that really enables a tester to get more efficient with taking mundane, repetitive tasks off the plate from their sort of commitments. And then once that's automated, it actually opens up their capacity to really do more of the real testing where the value of us as humans and us as testers can be more applied in raising awareness of the current state of the software as well as raising its confidence once it's finally out.

So I love that line, Torchbearer of confidence. That's probably the title of this Episode.

As a torch bearer of confidence, then.

I'm concerned, as we're moving rapidly towards continuous testing, continuous deployment, that more and more of these processes are getting automated, which is great, but how do we have confidence in our automation then has given us, giving us that confidence, the automation results themselves.

I think the way you approach automation really, and how you leverage that to ultimately build the confidence of software, I think the key is in there. I always tell my teams and I educate people who have just heard about the catchphrase saying that automation seems to be solving all the world's problems. Talking to my prospects who are asking for help in.

Improving the confidence of the software, I tell them that the one commitment you have to give when it comes to automation is you cannot treat it as an afterthought and do it only when your team's capacity is available or if you are not under hot water, if you will, under pressure to deliver. You have to treat any automation project as a software development project, right? You cannot come in and say, all right, I'm going to give, you know, commit to a certain capacity of the team to go execute on it and then completely revoke from it and say no, now that we have other more pressing issues, we're going to put this on the back burner. Every time you come in with that mindset, you are not going to achieve what success other teams and other delivery teams essentially have been able to achieve. So that's like the first thing I always tell my prospects and teams is you have to have a dedicated mindset. You have to be very clear on what goals you want to identify. Speed is something that obviously is of the essence and that can be achieved in one of many ways. If you go into an application or a team where they already have a pretty established product, the best bang for your buck usually comes in when you identify the high priority from a business perspective, areas of high touch, or if it's an area where it's high revenue generating areas per se, or what have you focus on those areas. Look at it as a tester. What are the key assertions and validations that you need to have to make sure that the system meets a certain expectation and then go ahead and then go look at strategies of how you convert that into automation scripts. Again, when you do your automation scripts, like I was saying, because it is an engineering project, you have to apply the same considerations you would when you write good code, right? You have to adhere to clean coding practices. You've got to have good naming conventions. You want to follow a Style guide, things of that nature. When you, when you engineer your solutions, that's the best bet of how you can achieve your success. Now, along with that, another aspect that comes into play is another end of the spectrum is where you come into a new product. And when you are launching a new product, you want to be able to go to market as quickly as possible, whether it's an MVP of your application or just one integration piece or what have you. At that point, that also becomes a great candidate where you say, hey, I want to start with an automation first mindset. Let me look at what testing and validations I need to have to be able to say that this application meets a certain confidence level or a threshold and then look at what components of that can be automated, whether it's data setup, whether it's environment setup, anything that helps me as a tester to get over the more labor intensive activities. I usually recommend my teams to push that into the overall automation bucket. And then what that essentially allows me to do is come up with a sort of a solution rather than just a core simple subset of activities which are automated and that is limited to just a functional validation of some sorts or UI validations of some sort. Does that make sense?

It absolutely does. So earlier when you were answering that, you mentioned something about it's almost like a whole team approach that in their sprint they need to include automation. And what I've seen though is teams, at least teams I work with, they get a feature and then scrum of scrums they go, oh, we didn't complete it because we had to do this extra automation activity. But it's actually automation is development, it's part of the story. So like, do you build into each sprint like a percentage of times dedicated to automation? How does a team plan for automation?

So typically for me, what I would do is I would put in the effort. Just like developers are expected to build in their unit tests. I actually work with teams that when we go in and we estimate the user story points, a critical factor of that is to say, hey, here's what the effort is to test it. And when I'm thinking of that effort, part of my tasks is to actually go in with an automate first mindset. There are certain ways to actually overcome the initial hurdle most team members have, which is, well, I'm going to test the feature first manually and then come back to automate it versus we looking at it to say, all right, let's look at this particular feature.

What is its different makeup? It has a UI component it has maybe a middle level API component and has maybe a data component. What from a tooling perspective that I can leverage now help me overcome that pre setup, pre prerequisite effort of having to even perform the manual test. Can I dissect my scope of tests in a way that I can go in with the tool first and have that mindset of saying, hey, this is something that is going to be repetitive. This is something that I will always need to do going forward. That gives me a better call on saying, hey, this is something that I should automate now. The risk that us as testers face is as we keep pushing the automation piece on our backlog. Chances are with the great demand of business needs within an application, you may not be able to get to a solid state where you may be able to go back and then add automation as an afterthought. My perspective on leveraging automation is always try to go with an automation first mindset, just like any developer would by saying, hey, I'm writing this code. It is also very important for me to create unit tests now that I can incorporate as part of my delivery lifecycle. We should have the same mentality as testers belonging to that same thing.

Absolutely. And I love the concept of having people think of the different stages that they could automate at. Because if you're not creating the code to be automatable, then if you are trying to go to continuous deployment, you're going to have a backup. Right. Because you want to minimize the amount of manual interactions, I guess, as you.

Absolutely. And you know, unfortunately, when we don't shift that mindset, what also happens is it gives and becomes another excuse where testing is then again considered as a bottleneck. Right. I mean traditionally that's how, you know, in the, in the pre Agile state, during the waterfall methodologies. Right. I'm hoping not a lot of people still use that, but I remember distinctly when I started my career it was always about, hey, testing, you're the bottleneck. How soon can you finish testing? And my discussions will always be about I can finish testing in one hour or one day because that's the level of confidence you will get given the time I get to spend on it. So tell me how much time can you allocate for me and I'll tell you from a risk perspective what coverage you can get. So this mindset of continuing to push automation onto a back burner and saying, oh yeah, we'll get to it either at a later sprint or at a later time, unfortunately just keeps being added as Almost like a technical backlog or technical debt.

Chances are the only way you can actually overcome it is when you have an external capacity coming, either through a partner or contractor who comes in and their sole focus is only to convert your sort of manual scenarios or tests into automation scripts. And again there's, you know, I think there's. That's another topic that that gets highly debated also is really is just converting your test steps manually into automation. Is that really what automation is all about? And I think that's a completely topic in itself as well to discuss and see what are the best ways of doing that.

Yes. So is that what it's all about?

So my philosophy on it, quite honestly Joe, is it's not right for me. I always look to automation as an enablement tool. So I would always question members or my fellow testers who would say I always do a one to one conversion of an automation script that is created from one automation test case or a scenario. And my point is, well, maybe that's not the most effective and efficient way.

The code that is written as an automation script should have its own sets of almost, I believe as KPIs, it should be able to run a certain way, it should follow good design principles and things of that nature. It's not necessarily should be tied to just a one on one ratio of saying, okay, if I have one test case, I just need to have one test script. You can get smarter about that and use object oriented methodology methodologies and things of that nature where you can actually.

Combine a lot of manual assertions that you do perhaps in 50 different test cases into a few test scripts.

Great advice so much. We mentioned continuous testing a few times. So folks that don't know, what's your definition of continuous testing?

So continuous testing to me is that methodology or mindset really where every time I am given a piece of code or a piece of functionality, I can quickly gain or assert its confidence level in its capabilities. For me it's an activity that can be done on demand, really. And those trigger points typically come in when code is checked in or in a state that it can actually run through its churn.

Perfect. So we did talk about you want to automate as much as possible and not automate just for automation's sake. You want to automate the right things at the right level, but you can't automate everything, can you? Like, are there any manual interactions that are unavoidable when you're doing continuous testing?

So I agree with you that not everything is automatable. And really just because it can be automated, maybe there is not an ROI to automate. Right. I think my rule of thumb that I personally try to follow a lot is to say look at it from a business perspective, look at it from a priority of what is the most important thing, and then build that with better engineering practices of saying, hey, how can I, if I wanted to get more confidence by running more test cases or more different variations of asserts and validations across a wider spectrum of the application, how can I actually execute in parallel? So there's really not one scale that you could go by to say here's the best way of doing it. My approach typically has been I look at the business impact of the different areas and as I understand the domain, as I understand the system, talking to the business members, say, you know, from a particular vision or goal of your application, what are the absolute must have areas by priority. And I try to push those as soon as possible, as early as possible and improve, introduce those as my key candidates for automation and then work my way down that list.

Perfect. So Masha, another issue I've seen is automation gets blamed for everything. But a lot of times if the teams are going towards continuous integration and continuous delivery, they're lacking in other areas like infrastructure. Like, you don't have an automated deploy, how am I supposed to automate? Or I don't have machines that can be provisioned automatically. What infrastructure is needed in order to be successful with automation in a continuous testing type of environment?

So I think speed again, I think becomes the bottom line, right. So every time I can have that advantage of speed and reduce or remove human error altogether, that would be my go to. Right. The way I like to think about it is, and I think this is what I would recommend other testers to think about also is as a role or as a member of the team playing the tester's role, I would think about what are the capabilities that a tester can bring to the group. Is that capability saying, we are now ready to deploy through containers? Are we ready to deploy in the cloud? Think of what features you want to be able to give as part of your automation strategy as these capabilities and then just work your way independently to be able to, you know, as part of your implementation strategy. Think about if you were to try to be able to deploy your framework or your lines of code in an environment other than where you are currently, and it may be the traditional one where you're in real environments behind a firewall or whatnot as you are writing code is it compatible with the next set of the DevOps toolchains. I always encourage my teams to think about it to say if you were to.

Spend 30 minutes upfront when you're creating a script or when you're adding capabilities to a framework, would you rather spend 30 minutes of extra time to see what you need to do to be able to have this adopted into a container for example, or would you be able to integrate into the developers GitHub repository just so that way it is already capable and ready to allow the infrastructure team or the management team to come in and decide to say hey, we want to switch the platform or we want to apply new tool chains. How soon can we do that? And the from a planning perspective, if you think about capabilities in that fashion when you're actually building your framework, it helps you make that jump even quicker. When I'm thinking of solutions, I'm looking at it from the infrastructure perspective also. The core driver for all of those decisions is all about repeatability with high speed, which means tools need to be a key part. Then the second aspect of it is being what task going from concept to live. Are there opportunities for me to remove the human repeatability and sort of dependency on human effort AKA which would also mean human error. Where can I eliminate those successfully? And that could spread across from infrastructure to repositories. The way you write code style guides things of that so much.

What are your thoughts on testing and production? It seems to be a bigger trend lately, especially as you go towards continuous testing. Any thoughts on that?

Yes. So I think to me an ideal state is where you have enough confidence in the code that's being pushed through the pipeline that you have it where it can actually go into production. What I have observed is there's a couple of ways how typically people do it. One is where you can almost deploy it into a certain cluster area of your live environment where any of the testing and activities that you do don't necessarily impact the other areas or existing features. So it's not breaking up activities that real users are doing. The other aspect of it, I know I'm a fan of.

Chaos the community and using tools again to inhibit behaviors and see how life systems react. The other aspect I've seen always is also using very distinct data sets that can clearly be isolated from downstream systems in production. I think the nervousness factor of what traditionally testers face where the production environment was a very scary environment where you have to be very careful and not be as.

Exploring to try to see if everything works I think that mindset is definitely changing. And I think a lot of that also has to do with the way code gets successfully deployed in production. If you are in a production environment that is actually on containers, that changes things where you can actually tear it down very effectively. But I'm a fan of that. I know where it is something that you should definitely keep in mind because that's the next sort of maturity iteration as your code gets deployed to production on demand or continuously.

Absolutely. The two.

Objections a lot of people bring up is, as you mentioned, test data, how to isolate test data from real production data, and also how to isolate or avoid introducing performance issues during your test. Any tips on how to get around both those objections?

So I think for performant aspects and introducing features and validating that without impacting the other areas, I think one of the areas where I've seen that work very successfully is deploying only in certain sub clusters of an environment and then keeping it just within that smaller area. I think Netflix is one that does that very, very effectively. You know, as an organization, they do a lot of testing in production. To be honest, and me being a very active member of Netflix, I hardly have ever encountered any issues that I encounter when I'm using their service. So that tells me it can be done. They demonstrate it can be done. And I think the way they do it is where they've actually been able to isolate it into a smaller cluster within the bigger production environment. As far as test data goes, there's a couple of ways of that also. If you are using container based deployment, then that makes it easy for you to quickly tear down all of the entries and items that you've added. Another component of it could be where you could have isolated users that are tied to specific data formats.

Prefixes, things of that nature. And then the key part in all of that also becomes communication to other members downstream who may be using data from production. Right. For reports and integration with third party tools and so forth, making sure that there are filters that are communicated and explained to those teams. Also to say, hey, can you add in filters to actually isolate these patterns of data could be another way of preventing any fire alarms.

So much I'm just curious to know. You work with a lot of companies, I believe a lot of different groups. Do you see any traits for a team that's successful with automation compared to one that's not? Maybe. Are there five pillars of a successful automation implementation as opposed to people that are struggling with automation?

Absolutely. I think the most important thing to me that makes the success of an test automation initiative is the culture of the organization, right. And how well the team adopts to that culture. So by culture, I mean the level of collaboration that must exist, right? Any, any automation project, it, it is and it is driving an outcome for the overall team. So collaboration between the product owners and the business analysts with the testers, collaboration of the developers with the testers, operations with the testers, they have to have good collaboration, good communication, good transparency in the activities. That's one aspect. The other aspect that I always see and count as a key success factor is being able to communicate the goal clearly and effectively. As I was mentioning earlier.

As a tester, you have to make sure that you're communicating to the bigger team that automation is a software development project. Don't make it as a side project.

Just do it when you have the chance, put it front and center. That's the other component. The third component of all of this is also building confidence in the automation that you actually apply. So teams that don't share their frameworks, their scripts with development tend to also be still continue to be left in the silo, be open to collaborating, to getting feedback, and then also recommending how our solutions can be leveraged by the developers. One of the key things that I've seen as a core success factor is as you build the confidence in the framework and the solution of the testing that you do with the developers, and you see them adopt and use them more and more, the easier it will be to actually collaborate with them because they will see, they will understand what you're trying to do and they are actually becoming your end user for the product that you're creating, your test automation framework. So if they are able to leverage that as well, it actually promotes a lot of collaboration and a lot of camaraderie, in my opinion, which is really, really good for the overall success.

Okay, much. Before we go, is there one piece of actionable advice you can give someone to help improve their continuous testing, continuous delivery, or automation testing efforts in general, and let us know the best way to find or contact you. Sure.

So I would think the best way is open collaboration and discussions with your team. I think a lot of camaraderie with your fellow developers and exchanging ideas of how to test better, test differently, think outside the box with a solution that is focused on speed as well as confidence. I think that's the underlying magic formula for me. And of course I'm reachable on Twitter. My handle is Mushonda. I'm also available at my company email. It's mush honda Ms. Technology.com thank you.

Mosh for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testtalks.com 262 and while you're there, make sure to click on that Sign up for our free trial link under the Exclusive Sponsor section to learn all about Source Lab's awesome products and services. And if you like this show, if you haven't already, why not rate and review it on itunes? Ratings go a long way to help boost the rankings in itunes and give us exposure to more people. So more people so you can help spread automation awesomeness even more to the masses. So that's it for this episode of Test Talks. I'm Joe and my mission is to help you succeed with creating automation. Aw Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com for full show notes, amazing blog articles and other automation awesomeness.