233: A Practical Guide to Testing in DevOps with Katrina Clokie

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of Test Talks. As organizations shift to a culture of intense collaboration and rapid delivery, the expectations for testers are changing. So what does testing look like in an environment with automated builds and deployment pipelines? How does the appetite for risk change once a product can be tested in production? Who should testers look to connect with across the organization? And how can they work together to effectively deliver software quality? All these questions and more is what we'll be covering today in this test talk with Katrina Kloki, author of the book A Practical Guide to testing in DevOps. As always, Katrina delivers the goods at a way that targets people and teams that actually make up the culture that needs to be in place in order to be successful with any type of DevOps automation efforts. She really gives awesome guidance in this area, so really appreciate her and this awesome resource. Listen up. And speaking of resources, another one you should check out, if you haven't already, is Automation Guild. If you're still looking for the perfect gift for that special automation tester or automation team in your life, give them the gift of automation awesomeness. This year. Check out Automation Guild, an online conference dedicated 100% to helping you succeed in creating automation awesomeness. Head on over to Automation Guild and register. Today.

Test Talks is sponsored by the fantastic folks at Sauce Labs, the cloud based automated testing platform that eliminates the need to maintain your own selenium grid and test infrastructure. Try it for free today. Visit testalks.com and click on the sign up now link under the homepage sponsor section.

Hey Katrina, welcome to Test Talks.

Thanks for having me back again.

It's great to have you back on the show. I think I had you over two years ago. Episode 111.

It's funny because I was thinking, oh, it feels like not that long since I did it. And then I went and looked it up and I was like, oh gosh, that was quite a while ago.

I can't believe it. So besides writing this awesome book, which we'll talk about a practical guide to testing and DevOps, what else have you been up to since then?

All sorts of things. So, a few different conference speaking engagements, we've run another couple of WHE test conferences. In that time I have picked up a really big project at work around improving our test environments. And outside of that, I've been doing A bunch of volunteer stuff with my guiding unit and with my Altruza organization. So it's been pretty busy.

Very cool. So first off, why did you write the book A practical guide to DevOps? You seem very busy, so it's very time consuming to write a book. Why did you put the effort into writing this book?

The main reason was that my organization were talking a lot about DevOps near the end of 2016, and it was a direction that they wanted to push towards. It was something that there was a lot of information for testers online, but you had to go and find it like it was all there, but you had to.

Know where to look. And what I wanted to do was to be able to give something aggregated and.

Kind of curated is the word to my team. So to be able to give them one thing and say, hey, you know the DevOps thing that everyone's talking about and it's making you a little bit nervous. Here's kind of a guide to what that means and how it might change the way that you're working. And the timing of it was kind of convenient. So at the end of 2016, in New Zealand, in the city where I live, there was quite a large earthquake and it had impact to my organization and that our building, our large corporate office building was shut down. We were all removed from the property, which meant that we were working remotely for about four months. So an organization of about 500 people just scattered back across the city. And for my role, that was really challenging because I work across a number of different teams and with a bunch of different people. And trying to do that when you're geographically dispersed is really hard. So I ended up with a bit of time. So my. My day to day work was relatively light. And what I ended up doing was in the evenings kind of, I was writing the book and that was almost the way I was keeping myself engaged. And my brain switched on during that period where work was very unsettled and people were spread all around the city and things weren't able to progress in the way that they normally would. So, yeah, that's where the book came from. A New Year's resolution, An earthquake and organizational strategic direction.

So that's really wild. So while you were writing the book, were there any particular areas that if you had implemented already in your company, would have helped you deal with that earthquake scenario?

Oh, that's a good question.

Yeah, I think a lot of the stuff where it would have smoothed it out for us was in that relationship building. So nothing to do with the technical solutions, but more about the ways that we connected across the organization because a lot of that actually happened in person because we were all in one place in a really large building. So the main thing for us would have been in our communication. So, so we were all co located in this very large building. And it meant that if you wanted to get anything done, you could go and shoulder tap people. And there was very informal networks and.

Kind of spaghetti communication where you know one person and then they would know someone else and then they would know someone else and then it would sort of end up with the right place. And you can get away with that when you're co located because that chain happens very quickly. Flow on effect of people walking to each other's desks. When we were distributed, suddenly everything slowed down and no one knew the end points that they were trying to get to and you couldn't shortcut that process in the same way. So it was actually really interesting seeing how quickly people became isolated from each other without being able to kind of walk around the floor and figure out who to ask. So yeah, it was definitely an interesting time. Actually it was a couple of years ago now.

Very cool. So this might sound like a strange observation, but your book to me is very people centric in that a lot of the examples you have are team oriented where you have people working together to actually come up with a, a plan for DevOps and a group agreement. Is that your approach? It seems to be that's your approach to most things with testing. Could you just talk a little bit about that?

Absolutely. So it is my approach and I also believe that is the ethos of DevOps. So I see in the industry the rise of the DevOps engineer and the placement of all these responsibilities and aspirations for DevOps coming to an individual. And I don't believe that was the intent of the movement and the principles that people were trying to achieve. For me, DevOps is about that culture change and the relationships and the people. And if you are focusing on tooling and individuals, I think you miss a lot of the opportunities for this to have a really big impact on your organization and the way that you work. So yes, definitely the focus of the book is on people and getting people talking together in productive ways, getting them to collaboratively agree on change, getting them to really understand the purpose of adopting a tool or a purpose of changing the way that they work so that there's that buy in and understanding. So yes, I am, I'm happy that theme came through for you and that you think that it reads as a very people centered book.

Awesome. And I guess along the lines, you start off in the first chapter on testing in a DevOps culture. And this to me seems to be the biggest roadblock in my experience with people. Like you said, they usually say, oh, we use cucumber, which is a tool. And they talk about all the tools they use, but they don't necessarily embrace the team mindset or the testing mindset. So you have some cool ideas on how to help that. Can you give some examples of how we can help collaborate better with developers within a DevOps type culture?

Sure.

For me, tested and developer collaboration is something that we should have been enabling in an agile environment. DevOps for me is about getting both those groups to go wider still and start talking to ops. But if we're just talking about culture within the team.

There'S a lot of practices that can enforce that communication until it becomes something natural. So I'm thinking about things like code review, testing.

Pairing on testing, or mobbing on different tasks, debriefs around your testing, anything where it's a practice that means people have to sit together and talk about or collaborate on something that they're doing. So even Three Amigos, or even any approach where you have people from different disciplines who have to exchange knowledge, it's a good way to start building the relationships. Because part of creating that respectful and collaborative environment is that you value what the other people in your team do and you understand what the other people in your team do. And those practices help that exchange of information so that you do have that respect for each other's skill sets. My observation of tester developer relationships that are really fraught is where often the developers don't respect or understand what the testers are doing. And that's partly in some cases, that's partly because the testers themselves are really bad at explaining or sharing anything about what they do. Like they're so accustomed to holding everything very close to their chest and it's kind of like, yes, it's tested or no, it's not. And.

There'S no sharing of what's going on when they're doing their work.

Absolutely. And I find that very frustrating as well. But another point you have in this chapter is, and you mentioned it, collaboration beyond development. So that assumes testers and developers, I think, already have their stuff together. What do you mean by collaboration beyond development?

Yeah, absolutely. So, yes, my assumption is that devs and testers already are already a unit when you're getting to this point. But this is about Reaching out to people who might not be part of delivering.

On your software platform, but they're more people who support the ongoing day to day running of that product for your customers. So if you're collaborating with the people who collect analytics metrics information about what your customers are actually doing on your platform, then that can help influence your test strategy because potentially you've been focusing on parts of the product that people actually don't use that often. And you may not be aware that there's another feature that people actually rely on heavily. So being able to have that conversation, there might be people who are doing on call support and they control.

How people get alerted when something's going wrong and what monitoring dashboards that they have available to try and diagnose problems. And potentially there are things that you prioritize in your testing that are important to the function of the product that these people might not have in their dashboarding and alerting. And you can have that conversation about why is it that I think this is really high risk and important for us to have rights, yet we're not even looking at that. And once the software hits production and vice versa, like maybe there's something that they're alerting on which you hadn't ever considered as being a core function. So it's just about having those wider conversations and for those people to understand how the information that they have can help you and how what you're doing might influence the information that they're collecting or using in their roles. So reaching out kind of across that barrier between delivery and ops.

Very cool. That's the first chapter, one of the first chapters, testing and DevOps culture. Once you have that culture in place, the next chapter is testing and development. I guess the biggest thing that jumped out to me in this chapter was bug bash. This is something that's been very beneficial to my team. It's funny because they always, once again emphasizing unit tests, automation tasks and tooling. But we tend to get the biggest bang for our buck is when they have these, we don't call them bug bashes, but we just have everyone go on and do exploratory testing on the system. So what is a bug bash and why do you think it's so beneficial to teams and DevOps?

So a bug bash is basically exactly what you've described, where you get everyone onto the software and exploring it themselves. Where I think interesting in a DevOps culture is that you can bring people from outside of the team into that practice. So bug bash, I'm going to say traditionally but it's not a super old practice. But Back Bash would usually be your development team. So your testers, divs, maybe your BA product owner, potentially like a scrum master, those sort of people who are close to the delivery and development of the software.

All of them getting hands on with the product, exploring the different features and trying to find issues or problems. And the thing about that is all of those people have some understanding of what you're trying to do. Like they all have some bias towards how the software should behave because they've been part of the conversations and they've seen the way that this thing has evolved as it's being developed.

They still bring a fresh perspective, but they're also still people who know what's going on. And when you're in a DevOps culture and you start to bring in those people from outside of the delivery team into those activities, you get a different perspective on the types of issues that you find. So for one, those people haven't been part of any of the conversations about what the changes or what the new features are.

For two, they have a really different understanding of the software and the way it works in general because they look at it through a different lens. They're looking at it through the lens of does it work, how it usually works, Is the behavior of this consistent with the way it's been in the past? That lens, as opposed to here's the requirements of our change, can reveal some quite different problems. Those people also have different ways of.

Monitoring whether something is successful. If I'm testing it as a tester, maybe I'm looking at logs that come out of the software. If I'm testing it as a person from an analytics team, potentially I've got a Google Dashboard open and I'm looking at how.

My paths through that application are being tracked because that's what interests me, right? If I'm looking at it from a perspective of someone who works in monitoring, maybe I've got my Splunk Dashboard open and I'm looking at how that's recording errors or issues because that's what I care about in the software. So it really broadens the perspective of the types of problems that you find because the way that people are thinking about it and the tools that they're using to assess whether it's correct or not can be quite different.

Awesome. So another point you bring up in this chapter is pipelines. I guess if someone's new to DevOps, they, they may have heard of pipelines but not necessarily know what it is. Are There certain activities that occur at each stage in a pipeline. I guess what is a pipeline and how does testing fit into a pipeline approach? In DevOps?

I think pipelines are relatively common and agile and that might be an assumption that's incorrect on my part, but my exposure to pipelines and agile organizations has been pretty good. So that traditionally in those environments about getting the software from a commit to release. So if I'm in that development team, when my developer has had their pull request or code change reviewed and it's getting merged back into our master branch, the pipeline is what takes that piece of code from that merge through to production. Ideally it's running a bunch of automated checks. It might be moving the code through different types of environments and deploying it and building it again.

And shifting it through that process and validating that it meets some things. There are challenges for testing in this where if the entire pipeline becomes automated, your opportunities to actually test, as in exploring the software hands on, get squeezed out like Play DOH to either side of that pipeline. So you can explore the software prior to it being merged and you can explore the software in production. If your whole pipeline is automated and things are traveling through that very quickly, it really shifts your opportunity to explore.

What I write about in that chapter is that in DevOps I see teams who extend their pipeline. So instead of stopping that pipeline at release, they actually push it further and start to run some tests in production as part of their pipelines.

It might not be all of the tests that you would run pre release. It almost certainly won't be. But it's choosing some of the tests that are maybe aren't writing data or like they're just doing views or they're validating existing content in your system. Potentially they can be writing data if you have specific types of labeling or something.

But it's. It's about getting the pipeline not to just drop it into production, but to actually.

Run some checks to validate that what you've dropped in production is behaving the way that you think it should be.

Awesome. That's a perfect segue because that was the next chapter. Testing in production. One of my favorite chapters, because you go over some cool techniques. I think once again, maybe I'm making an assumption that most people may not be aware of or use. I guess the first thing is once you're getting all this data, it's supposed to help your team, so you need to bring it back into the team. So how do you get actual feedback from your monitoring that you recommend to a current DevOps.

So getting the monitoring back into the team in my organization that generally happens by the development teams getting access to the tools that monitoring the ops people use.

For us, that's our Splunk dashboards or Dynatrace.

Some of our other tooling that specialized to particular databases where it becomes something that the developers actually have bookmarked and will refer to when.

Not just when things are going wrong, but also on a semi regular basis to see what's happening in production and to understand the outcomes of what they've released or changed. So they're looking at the ways that.

The measures of their success, how those measures are shifting. So most of our teams work towards a business oriented mission. So it might be I'm converting our nps, our promoter score, the customer satisfaction measure for our product. And the focus of my team is that and the actual features I do to towards that might vary but it's that measure that I'm concerned with or home loan conversion rates, who's putting fixed rate interest on their home loan. And that's my metrical measure. And I want to know I can do various things to try and drive that outcome. So if I've got access to monitoring information from production, I'm looking at that to try and understand how did the most recent change we made, how are people using that in a positive or negative way and how is that impacting on the business outcome that we're trying to drive towards as a team? If I'm not engaged in that production information, it's really hard for me to understand the impact of what I'm doing towards that mission that my team has. So we've put quite a bit of work in to try and pull the development teams away from getting given a feature. And they now have a much broader understanding of how the changes they make to the products correlate towards what the business is trying to achieve and how to understand progress towards those goals by using some of the operations tools that are available to them now.

I guess another concern people have is actually testing in production. But you recommend a few approaches. One of them is called Dock Launching and I'm not sure if it's the same as Feature Flag. So can you explain a little bit what Dock Launching is, how that can help test a feature actually in production without impacting customers?

Yeah. Yes. Dark Launch and Feature Flag. So Feature Flag is something that enables you to do Dark Launches.

So you can put put code into production that the users cannot see or cannot interact with. It might be that if they knew the URL to go to. They could see something different in a web app, or it might be that there's a script or console command that you can run to jump into some other feature of the product, or a user based toggle that you can put on for someone so that they can see something. So the idea with this is that.

We want to be pushing code to production frequently, but we may not want the users to see those changes at the same frequency. We're trying to reduce the risk in us pushing out a large piece of code and seeing impact from that that we don't expect. But we're also trying to mitigate the way that users respond to change and prevent them from getting fed up with things always shifting around or being slightly different to what they expect.

It's an approach to assist both the developers and the users of the platform to reduce the risk in us making really big changes, but in some respects still retaining that big change feeling for the people who use the platforms on a regular basis.

Awesome. I'm running out of time. I like to keep it to 30 minutes. But the last chapter, or one of the last chapters was testing in a DevOps environment. What I've seen a lot is teams struggling with environments, which takes up all their time just to set up a valid environment. Environment. What is the role of environments? Like, what do you mean by testing in DevOps environments? Are there any tools that can help someone with the infrastructure to help make testing easier?

Yeah. So my purpose with this chapter was to try and move people away from the idea of having a dedicated test environment. So as testers, I think we've got in the habit of being the guardians and defenders of these machines and databases that we've set up in a certain way. And there is no way that anyone else is going to touch that because that customer in that environment is now in the perfect condition for me to do what I want to do.

People from a DevOps world talk about servers as pets versus cattle. So they're saying we used to really love and look after our hardware and our infrastructure because it was like a pit, we only had one of them and we needed to keep it, you know, healthy and watered. And now we're moving to a model where cattle, it's like farming. We have lots, they can come and go. We're not going to name them potentially and love them and look after them in the same way. They become something a little bit more disposable. And I think that shift in mentality is something that has to occur in testing as well, that you can't sit with these environments that are the same all the time and you guard with your life. It's more about adapting towards that culture of cloud based virtual infrastructure that spins up and tears down as you go. You might be ending up using multiple different environments through the day, which could be images or duplicate copies of the same baseline.

So you still can have that consistency in state, but it's different infrastructure that's providing you with that environment with which to test. And so there's some interesting opportunities with that where you. It's easier to kind of reset and you don't have to be so protective because the infrastructure enables you to get back to that state that you were so precious about. You don't have to prevent other people from being in your environment or doing things in order to keep that same.

Yeah.

Keep that same state or keep the platform in a way that is useful to you.

Where can someone get a hold of a practical guide to testing and DevOps? I know it's really highly rated. You're getting awesome feedback from it. You've been selling a lot of them. Is it available on Amazon? Is it just LeanPub? Where could someone get ahold of your book?

At the moment it is just on LeanPub. It's just an ebook and when I wrote it, I was always intending for it to be an ebook. It's very rich in references, so there's over 170 links to other information across the Internet. So an ebook is the perfect format for it. And on LeanPub they're really good to their authors and so it's a great platform for me to provide it through.

Awesome. Okay, Katrina, before we go, is there one piece of actual advice you can give a team to help improve their DevOps testing efforts? And let us know the best way to find or contact you.

Actionable advice.

If you don't already know what's happening in your production environment, that would be my first piece of actionable advice. Either talking to someone who supports your software in production day to day and or trying to get access to the monitoring tools they use to figure out what's happening.

That's an excellent first step. Just trying to build. Build your own awareness of what happens after you're finished with this thing. Like what are your users actually doing and how do the people who support your software understand if it's behaving correctly or not? And just developing that understanding is a really good first actionable step. Contacting me? You can contact me via Twitter or LinkedIn.

Yeah, awesome. Thank you so much Katrina. Thank you so much for this resource because it really does tackle to me what is the hardest part of any sort of development or testing and that is culture. This book, like I said, is really people centric and really team friendly. Thank you for that. Just really good spin on it. So thank you.

You're welcome. And thanks again for having me.

Thank you Katrina for your DevOps testing awesomeness. The links to everything of value we covered in this episode. Head on over to testtalks.com 233 and while you're there, make sure to click on that. Sign up for a free trial link under the Exclusive Sponsor section to learn all about Source Labs awesome products and services. And as I mentioned, why not also attend Guild conferences to really up your Automation game in 2019. Head on over to automationguild.com and register today. So that's it for this episode of Test Talks. I'm Joe and my mission is to help you succeed with creating automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com for for full show notes, amazing blog articles and other automation awesomeness.