How Quality is an Integral Part of Digital Transformation with Chris Rowett

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Chris Rowett, all about how quality is an integral part of digital transformation. Chris is the Vice President of Customer Success at C Lights. He is responsible for helping organization organizations adopt a truly transformative approach to delivering high quality applications faster at a lower cost. If you don't want to get left behind as more and more companies start transforming the way they do business, you want to listen to this episode, Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial, visit guild.comforward/source labs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Chris, welcome to the Guild.

Hey Joe, it's great to be here. Thank you for inviting me.

Great to have you. Before we get into it, Chris, is there anything in your bio that I missed that you like the Guild to know about?

Yeah, Joe, thanks very much. So, yes, I guess over about the last decade I've been really focused on the challenges of digital transformation. So I've worked with many enterprises around, around the world. I've worked with kind of CIOs and their teams on really thinking about what does digital transformation mean in terms of a process of change, touching organizational design, touching automation, agile, DevOps and ways of working and ultimately then you know, the tools that accelerate that. And it's those learnings that really helped me gain the insights to, you know, help organizations understand how they prioritize these change and you know, we'll come on to talk about it. But the culture of changes is hard and so providing organizations with the insights they need, the data they need to make the decisions that keeps driving quality change becomes critical. I think one of the things that was interesting recently is, I mean we're going to talk about digital transformation today and in some of my spare time I play in a worship band at a local church and with COVID and all that stuff, we've now moved to working at home, but we've also moved to doing everything else at home. So we're now doing music digitally remotely. So you know, we've gone through rather than stand together on a stage and rehearse and play, we're now doing that distributed Remote bringing it back in, using new technologies. So it's been fascinating how digital transformation and Covid and other things, you know, pervaded every aspect of life. I found that fascinating.

So that is interesting. So with COVID 19, do you see digital transformation accelerating even more then? Because that's a great example. A lot of churches around where I'm located are doing the same type of setup.

Yeah, I think definitely digital transformation, I mean obviously the digital transformation journey is not new. It started several years ago really by, you know, as a reaction to new digital disruptors coming into different markets, different industries and those sorts of things. And what it really impacted was the larger enterprises because they necessarily didn't have the agility, they didn't have the modern processes. They had tons of complexity, different infrastructure, all the way from modern to mainframe. So their ability to change fast and react fast to new digital business models was putting them under a huge amount of pressure. And so the growth of digital transformation was born. The key being the T word transformation. I guess for decades these organizations in IT had been fine tuning their approach to specifying, planning, building, testing and releasing software. And all of a sudden that was not good enough. But all of their people, all of their processes, all of their tool were focused on that, well, trodden path. And all of a sudden to either survive in their markets or to become the disruptors themselves, they've got to completely rethink what that looks like. And to your specific question, Covid's put another dimension on that because now those teams that we've been modifying and changing and maybe we've been adopting new ways of working, we're becoming more agile now we're distributed. Right. And now we're having to use more digital communications to collaborate. This is definitely driven a rise in the need for better metrics, better understanding across our software processes, where we are, what we're doing, what our quality is and those sorts of things. So yes, the short answer your question is yes, it's definitely increased the pressure on organizations to be more deliberate about what they're doing and how they measure it.

Absolutely. So you know, I was speaking when we were able to go to on site conferences with someone and they mentioned, you know, no one uses the word digital transformation as a tester. It's usually like a management type of term. So how would you explain what digital transformation is at a high level to like a practitioner? And then I'd like to dive into each piece that makes up what digital transformation means.

Yeah, great question, Joe. So I guess as a tester we're a key part of the end to end value chain around our organization, delivering value to our business partners and customers through software, right? Through web, mobile, APIs, core systems and those sorts of things. So the number one focus on digital transformation was the ability to do that faster and at better quality. So in this world, we live and die by the quality of our end user experience. So quality really became an integral part of digital transformation. We could argue it's the number one goal and right behind it is speed. And unfortunately right behind that is keeping cost flat. Right. So historically, if you think about the quality speed, cost triangle, you know, if we wanted to go faster, we could throw more money at it or we could make some shortcuts and reduce quality. Digital transformation means we have to go super fast. We have to deliver an outstanding end user experience at high quality. So now we're thinking QA specifically and the quality function. It's become a first class citizen in that whole process. We need to be thinking quality about everything we do in that whole end to end process. So in some ways QA has invaded the end to end process and we're having to think in new and innovative ways about how we do that, how we manage end to end quality and what our organizations look like to support that.

So one of the pieces of digital transformation that you mentioned was improving end user experience. How does digital transformation help improve the end user experience?

I guess two ways. The first is our ability kind of to put the customer in the middle of that process with the digital channels that we deliver through more than ever before, the feedback we can get on how we're doing, the quality of our app and those sorts of things. We get very, very quick feedback about whether that's working or not and whether we're providing a good enough quality of service, a slick enough service that's meeting their goals, and when it doesn't, we can measure, we can measure the bleed of customers, we can measure the bleed of revenue and these sorts of things. So that user experience is number one. It's our lifeblood. So how we keep focused on that becomes key. And when we're talking about quality, what are we specifically talking about? Things like you, production incidents. If I'm using online banking for a new bank, but their application doesn't work or it fails. When I'm trying to do simple everyday banking types of things because of quality issues, not only might I then quickly switch to another provider so they've lost my business, but the reputational damage that they might get if it's A broad enough problem, maybe they'll be on the news. Right. Maybe it'll impact their stock price. So now you think about the importance of quality in a digital world because it's so much easier for us to switch providers and change our minds and those sorts of things. If quality isn't outstanding, if the experience isn't slick and outstanding, we'll go somewhere else. Right. So we have to drive a quality first principle through everything we do and reimagine our QA functions to be supporting that.

That's a great point. Quality first. I used to work for a very large organization and they used to kind of rely on risk reviews. And the problem with risk reviews, it really relied on trusting what people were saying, but they didn't necessarily have a hard data to back up necessarily what they were thinking. So does digital transformation help people make better decisions, like when they should release based on better data as well?

Yes, there is a requirement for better data. So digital transformation means we will be building more automated pipelines. We will be trying to use that to accelerate how fast we can move code through the pipeline and get it tested, get it ready for release and these sorts of things. But what we add to that is what you're alluding to, which is quality intelligence. What metrics can we gather through that end to end process, through unit testing, functional testing all the way through to be making smarter decisions about whether that code is ready for release. Given what we talked about before, we cannot afford production incidents and we want to minimize those. So the only way we're going to improve that is by getting the right data at the right times to empower the decisions at those quality gates. So we have to be smart about making release decisions that has to be based on data, not on a gut feel. Right. So a lot of things we're going to do around digital transformation is in our pipelines be able to add the right tools that can surface the right insights at each stage of that process so that we can make those insights actionable. We can put them in to our quality gates for a go no go decision. We can prevent poor untested code going into the code base in the first place by helping people authorizing pull requests, have smarter data around how well it's been tested and those sorts of things. So yes, it's about data, it's about driving the data to the right places in the pipeline. So we are making data driven and fact based decisions. We're not just relying on our gut or how did it go last week and those sorts of Stuff. I guess the final thing I want to say there is, you know, in the last years there's been a big push on automation, right? CI, cd, continuous integration, continuous delivery, automated testing and those things, they're great. I mean they go fast, but they don't necessarily make any difference in quality. It's about, that's about speed. So unless we can augment our automation engines with the right data to make the best decisions, we're not actually improving the quality bar at all. So quality intelligence, getting the right data at the right place so we can make those decisions and we can feed those into our pipelines is mission critical to achieve quality and speed.

So it's interesting you mentioned quality in both your answers and I don't know what the saying was. I think you can only have two of either quality, fast or cheap. You can't have all three. But it seems like quality though needs to be in order for people to have a better user experience and be able to make better decisions, quality needs to be baked in. So I know you all speak to a lot of different companies. Have you seen more and more companies embrace this quality as everyone's type of responsibility mindset?

Absolutely. And I think that the biggest, the strongest message that's coming across is quality is no longer a single department. Right. Quality has become everybody's responsibility end to end, right? From how well we're defining requirements to how good is our testing and test coverage, how robust are our release processes. So quality is now a holistic thing. It's not a dedicated function. We still need the expertise of the quality professionals. We're providing it across the entire end to end process, as I said, in a holistic way. So what that's meant to organizations is some of them. And you've got the whole gamut. I mean, you talked about the fact you previously worked in very large enterprises, some of them, I've still got very traditional software lifecycles, software pipelines, testing processes, some of them have got areas of very modern agile pods, all these types of things. But whichever process they're adopting, whichever methodology, quality is now an end to end issue. And in the organizations we're talking to, it's being driven top down, right. So there are requirements from senior management to understand the quality bar we have in our organizations and to be able to measure it. So that's back to what I said before. How are we getting the quality intelligence, the insights out of our pipelines on which we can make better decisions about whether to release or not at the tax practical level, but then More importantly, as we're driving the quality agenda through our development, our qa, our release teams, how are we measuring their maturity? What metrics can we pull out? How can we look at untested code, code coverage, those sorts of things to allow the senior management to make better decisions about how those teams are ramping up, how are they improving their quality maturity if they go through what percentage of time should we be focused on improving our aggression suites and those sorts of things? So we're definitely seeing a top down requirement for better quality intelligence and then using those insights then, then in a drill down level to the various teams to say, okay, here's how, here's what we're looking at guys and here's therefore the metrics you need to be driving for in your development teams, in your testing teams, in sprint squads to make that better. So it's definitely become a top down initiative.

So I have seen top down, but I also seen bottom up. And with bottom up it seems like most people are really focused in on tactics. And so you have multiple teams doing type of tactics with no one really looking at the bigger picture that's going to be bubble up to the management on top. So how do you go from like all these multiple teams doing tactics to really doing more of a holistic type of view as well to look at the big picture as they're doing their tactics to get to that big picture?

Yeah, that's a great question. And you're right, I mean historically a lot of the innovation around quality has come bottom up from teams innovating. But they were solving the point problem, right, for their application, for their business service, for those sorts of things. If you want the organization to move, it necessarily does need to be top down initiatives. So it's got the power behind it, the drive the to make the change. And how you really have to do that is by providing a consistent and aggregated view across all of those teams. So things like a consolidated quality dashboard that's going to show you across your application landscape how the quality is improving, which areas do you need to continue to focus and how those things work. So that consolidated view allows those top down executives to continue to focus on the areas that need further development, where they need more training, where they need to keep investing in, how fast is the maturity of the teams coming up as it pertains to quality and specifically closing quality gaps.

Great. Now Chris, I always say this on my podcast. I don't know if people get annoyed with me. I really love speaking with vendors because I think vendors really have a better insight into enterprises, which I think have very complicated scenarios. And so if you can make it work in an enterprise, you probably can make it work at a smaller company. So have you seen any customers going through this transformation? Do you have any like real user cases you can share on how someone went through a transformation, what those priorities were or how that journey looked like? Or what that journey looked like?

Like, yes, certainly. One of the things I did in preparation for this is I went back through a bunch of my notes, recent customer calls, to sort of aggregate some of the hot items. And in all cases I'll speak generically and then I'll talk about a specific case. So in all cases, you know, the commentaries were all around that organizations are trying to become data driven in their quality. And so how are they going to do that? How are they going to get the insights to do that? And what does that mean in terms how they're going to change their software quality, governance around what they do? Because governance is the way that they can actually use the insights, the quality insights, how good is your testing, where's your gaps in coverage, those sorts of things to then provide metrics into quality gates into how teams are maturing up that curve. And so when we talk to organizations, they're always starting at that point. It's always a desire to have better insights, to allow them to bake those into their various development and test processes so they can enforce better behavior, they can measure how they're improving, and they can also let the teams know that this is key, this is important. The fact there's metrics, and we're baking it into these particular discussions and KPIs we have with you, and it's appearing on a monthly dashboard means it shows the importance to the organization of this journey. Now, in specific cases we've got customers who, from, as you say, from big to small. In some cases, what they're trying to do is build the modern architectures, so they're trying to reimagine some of their core applications in terms of microservices, cloud, native, all those sort of things, and they're using POD and modern agile development methods to achieve that. In others, they've got legacy systems that are 20, 30, 40 years old, which are still core to the business. So in both cases you've got a complementary but different set of challenges. So if you're doing, for example, a mainframe migration, how are you testing that? How you, you know, and maybe in those cases the original developers still aren't around the original Requirements aren't still around, but we still need that's the core system. So we can build APIs and other layers on top just to expose those capabilities to our customers and partners. But the quality of those legacy systems is just as mission critical as ever. Anything we're building cloud based and microservices based. So the same requirements around. Okay, how are we then understanding in those legacy applications when we do test or when we monitor production customer interactions, what code is actually executed and where our gaps are, how much production code today has no test coverage at all. So those kind of insights allow them to then work with their partners, work with their internal teams to understand where they're going to focus their investments to close the quality gaps first. So a lot of this is cost, speed, quality cost is super important in this calculation. Understanding what's the best bang for buck in terms of what should I focus on first? What are the high value customer transactions running today that we know have zero coverage? Because those are the first ones we're going to spend the time and money to build better testing, better regression tests and those sorts of things. So providing organizations that roadmap to close the current risk surface, reduce the risk surface by understanding where the big ticket items are totally missing coverage. And those are the things we're going to build out in the next few sprints as we improve our aggression pack. Combined with the second part, which is okay, maybe it's a greenfield new application or maybe it's a more manned application that we've got a stream of change that we need to provide to. We want to make sure that every new change that goes through has got appropriate tests. So we have to get as fast as possible to the stage where every new change to our applications we guarantee before it gets committed back into the before we approve the pull request. We know we've got tests that execute every change we've done. And that way on one level with the overall coverage, we can slowly reduce the risk surface of the existing code. But we've stopped it getting worse because new change by new change, they're only being committed, they're only being approved if there is tests that cover every new change. So we've got customers are definitely taking those approaches. Their next challenge then is really around adoption. So how do we yes, we've got tools that are giving you this insight, giving us the data that we can action but how are we ensuring that all the teams are actually using that? That's where quality governance comes in because then they put in the processes and the metrics and the KPIs that say you have to show continual improvement or the release won't happen and those sorts of things. So it's that next generation of insight that drives, drives the governance that allows adoption to proceed. Otherwise teams are too busy and they've got a hundred other things to solve.

So Chris, I've worked for a larger older companies like Assurance and Healthcare and a lot of times they really relied on things like a reactive type manual approach to measuring and helping them go through their transformation. I believe tools like C Lights have AI based technology built into them. Can this help them actually make the transformation smoother or easier for them?

Yeah, Joe, I mean 100%. I mean it's one of the core tenants that we bring in terms of value with the C Lights platform, as you're right, historically a lot of the work we did in quality was reactive. We're just reacting to results, test results, what passed, what failed and those sorts of things. One of the big pieces that we're bringing is around artificial intelligence machine learning in the platform. So we're using this to really inject quality governance into the process. So the intention is rather than being static based on a quality phase, we want a holistic view across the pipelines end to end. We're going to learn all of the testing, the test phases, the coverage, build by build. And then the AI machine learning engine is going to use that to make recommendations to us about where the gaps are, what we should fix first, how we can decide which tests are worth running and which tests aren't worth running. So it's not just about improving the quality bar, it's pushing quality end to end across the process. And the platform then is giving us the insight to both prioritize where we invest in closing quality gaps and reducing the quality risk. But secondly, we can use that same recommendation engine to be saying on a build by build, test stage by test stage basis, which tests from maybe the massive regression set are the ones we should run because there's maybe only a subset of those that make sense based on the code changes that just happened. So we can only run that subset. We can get just the same high level of quality at much faster speed. So back to our speed quality cost, that kind of AI based recommendation engine is really going to accelerate how much time. Well, sorry, reduce how much time we spend in the testing phase, but at the highest quality bar.

Absolutely. And I love what you said about smart test execution. I really see that being the wave of the future. For folks that haven't heard that episode I had with Aran Sher, check that out. Episode 267 Where We Talk about smart test execution. So Chris, besides that, is there any way. What I see a lot of times is cultural issues holding people back and I'm wondering can AI and machine learning help maybe make more visible team collaboration or productivity? Is that possible with this type of technology that C Lights might have or other companies like satellites?

Yeah, absolutely. And we touched on this a little bit earlier. So one area is definitely in collaboration and this has become a hotter topic now we're in Covid and working from home in these sorts of areas. We have these distributed teams. So the ability to have the platform provide you that ins not just for your development team, not just for your QA team, not just for one application, but across your whole application landscape, across all your test stages and provide you those consolidated views that you can share with the teams that you can review in End of Sprint reviews backlog planning that the management can see on a monthly basis. In terms of trend reports, how well are all the teams closing quality gaps? How are we improving our coverage? How do we reduce the risk? So having that end to end, cross build, cross application consolidated data that we can share at all of those different levels gives us insight and collaboration that we've never had before. And back to the digital transformation that becomes mission critical to meet those digital transformation goals of ensuring that we've got the right insights to make sure that the acceleration we're trying to do with automation is at quality and not just the first application, not just unit test but all applications across the end to end all test phases. So 100% the ability to aggregate and share those insights, insights at each level of the organization and make it actionable. So yes, I'm a QA manager, I've noticed that for a particular application trend report is showing me that in general the modified code coverage, the amount of code we check in that got test is going down. So that's alarming to me. But I can use the same data, the level of drill down and go to the dev teams and say well actually the reason for that metric is because we've noticed that the number of unit tests per change is decreasing. So for whatever reason, but now we've got something specific actionable that we can give to the particular development team to say here's how you correct that. It's that ability to pinpoint where the problems are and allow people in a cost effective manner then to quickly change and measure the improvements month by month as we go so your collaboration that it brings across the thing is probably one of the strongest points and it's been definitely exacerbated during this enforced work from home mode. We find ourselves all in now.

Absolutely. And what I love about this is when I talk about automation, everyone just thinks functional automation or test automation. What I love about these newer technologies coming out is, like you said, it's actually automating things that were more of a manual reaction, active process, but in an automated fashion. Not necessarily a functional test, but injecting quality using an automated type approach powered by AI machine. Is that what I'm hearing, what the future is or what the current may be with some tools like C Lights?

Yeah, I mean definitely that is exactly where it's going. We started by talking about automation being a great thing for speed, but automation on its own, if you don't think about quality, all it really does is accelerates the problem. We can push bad code into production faster that you know, the anti pattern of what we want to do. So adding to automation the AI, the machine learning, that's taking into a whole bunch more factors. So I guess today a lot of quality gates are really based on did the test pass or did they fail? So how much can we improve that? So for example, on C Lights we can also look at things like where are your test gaps, which code hasn't got a test that executed it at all? And maybe just we can also just focus on the modified code. So based on the recent check ins that have gone into the build we're now doing and we're now executing how much of that new code has been changed. But then the future, we can make automated decisions based on that. We can put that into our quality gates. We can use those quality gates either to fail a build or to notify and quality governance policies. But you can also imagine in the future we can use those same ideas and extensions of those same ideas to start thinking about smarter decisions. We can start looking at much better risk profiles. So perhaps because we've got the data that shows over the last history of end builds, how often did those builds succeed, how often did escapes defect based on the particular module or maybe the particular team? So we can start using the complexity of the code. So there's lots of in the future, there's lots of other things we will be able to build into the platform to feed the AI, the machine learning to make better informed decisions that lower our risk profile around how fast we can push code in production and how smart those quality gates can be. So C Lights, I think is the beginning. The engine we have and what it empowers teams today to do is huge. But the future is also bright because as you start adding all those extra dimensions to that, it gets more and more intelligent and it gets more and more informed about the decisions it can make to help you achieve the quality and speed.

So, Chris, one thing that I've seen with digital transformations, a lot of people talk about shift left, but there's also been more a movement towards shift right in production, which I think is really critical. Once you release an application in the wild that you need to have insight that you then use to feed your pipeline from the beginning to the end. Again, so there's any technology or features you can use, AI and machine learning to help people with production like issues or quality metrics that could help them get better with the digital transformation efforts.

Yeah, great question, Joe. And we've definitely seen the same shift left and shift right. So earlier when I talked about end to end visibility, that does go all the way into and including production. And at C Lights, we've got specific capabilities in production to be able to actually monitor and listen to the deployed application. So we can then give you the same coverage. We can find the same quality gaps based on code that's running in production even after it's been deployed and checked through the earlier stages. Now that's generally useful, but there are some specific cases which makes that even more helpful. So for example, we touched earlier on legacy applications. Now in some cases those legacy applications, there's no documentation, there's no people around, you understand them well, or these sorts of things. And maybe there's not even test packs. So in those cases, if we really want to reduce the risk, surface our option, maybe our only option is to actually track them in production. So the nice thing about that is this is the real customer interactions. So we can actually learn from how customers are really using the deployed app applications, which methods they're doing, which code paths they're actually using. And from those build, you know, the AI and machine learning will get, then recommend us in the right order the areas that we need to build the tests. And building tests is expensive and costly. So providing that kind of prioritized area, these are the critical customer interactions with the code. Here's the ones that have got absolutely no test coverage. So those are the priority ones that you need to start building regression tests out for. So yeah, end to end, right from, you know, the developers working with a unit test through to actually monitoring in production all of that data across the whole pipeline comes into the platform, AI machine learning runs against it. And so we get recommendations regardless where you are.

So Chris, I may have failed to mention I just made an assumption. I've had three episodes already with the folks at C Lights, but for someone just listening for the first time, we've mentioned C Lights a little bit, but they may not even know what we're talking about. So at a really high level, really quick, what is C Lights for the folks that aren't familiar with C Lights?

So yeah, so C Lights, our focus really is on helping organizations with improving software quality, governance end to end. So our platform is not a testing tool, it's not a CI tool. What it really is designed is it plugs in to your end to end pipelines to gather and provide the insights around the quality end to end of the processes so that we can make those insights actionable to quality and development professionals. So we plug into, we track every build that goes on, we track every test in every test stage and we use that information and that insight to allow better governance decisions to be made. We also, as I said, as we've spoken earlier, we also can track applications running in production. And really what we're providing is which code has been tested, which code has not been tested, what's the modified code coverage? And then allowing organizations at the enterprise level to use those metrics to track and improve the maturity of the quality of the organizations all the way down to allowing, for example, someone approving a pull request to have the right insight they need at that point in time to say whether that should be approved based on is there sufficient testing based on the changes they have made. So ultimately organizations are using us to solve the big questions around is our quality good enough? If it's not, how do we measure it, manage that, and how do we provide the insight that supports their quality software governance processes? What are the metrics that they're making those judgments on? So we're the platform that's providing them insight based on so they can make the decisions at those levels. And we do it through machine learning, artificial intelligence. So there's the recommendations we're making allow them to action those insights in cost effective ways.

Nice. Okay, Chris, before we go, is there one piece of actual advice you can give to someone to help them with their digital transformation automation efforts and what's the best way to contact to and learn more about C Lights?

So I guess the advice would be, you know, get started. And when I used to, you know, run digital transformation workshops, organizations spend a long time thinking about it. Digital transformation is something you can't afford to wait. It's now the changes for a good reason. We need to stay relevant in our markets. We need to move fast. So check out sealights IO all the information you need to understand how we can help accelerate that is there. You can request a demo and take things from there.

Thank you Chris for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com.

And while you're there, make sure to click on the Try Today link under the Exclusive Sponsor section to learn all about Sauce Labs awesome products and solutions. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to test visitestguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.