Accelerating Large Scale Digital Transformation with Dilhan Manawadu

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating automation awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast. And today we'll be talking with Dylan all about accelerating large scale digital transformations with automated test data and environment management, which is a huge topic, so don't know where I'm going to go with this, but he has a lot of experience. He has over 15 years of experience actually in software development, specifically with large multi international organizations which I love. He currently serves as the Director of Software quality Engineering and development Platform at Cisco Corporation Houston, overlooking enterprise quality engineering and developer platform teams supporting digital transformation initiatives at Cisco. So he really knows his stuff. He's a lifelong learner. He likes to keep himself regularly updated on topics such as agile software development and Delivery, Quality Engineering, DevSecOps, ALM Engineering, Productivity Practices and much more. So really excited to have him on the show today. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Lab, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit testguild.comSource Labs and click on the exclusive sponsor section to try it for free today. Check it out.

Hey Delon, welcome to the Guild.

Thank you Joe. It's great to be here with you.

Awesome to have you. Before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

No, Joe, you did a great job in covering that. Thank you.

Awesome. So to start with digital transformation, it is a buzzword. So I'd like to start off by just getting your definition of what is digital transformation.

Sure, Joe. I think digital transformation has multiple aspects to it in terms of where we are in the industry 4.0. So today we have come to a digital world where we have so much cyber, physical systems, Internet of things, networks connecting together and if you look at the history of Industry 3.0, Industry 2.0, Industry 1. So the overall idea was we came from, you know, power weaming looms to mass production, electrical energy, automation, computers and electronics and the age of oil. And now we are in the age of data. Right. So today's fuel or the oil is basically data. So in my perspective Digital transformation is using this data and taking it to a different scheme of things and making organizations be digital first rather than, you know, taking digital and data as afterthought. So that's my take. And transformation is basically where large enterprises such as Cisco and Fortune 100 companies moving from this previous era with legacy applications and products to the current aspects of, you know, digital world. So that's what I think of the Joe.

Great. So I don't know if this relates at all, but I've been hearing a lot about Web 3.0 with crypto and not crypto but blockchain and technology like that. Does that fit into digital transformation or is that a different part of a company's, you know, evolution over time?

I think it's an evolution over time as well because it depends on the industry and where you are in the industry in terms of maturity. So I would say it's all part of that and it's all the future. Some organizations must be faster. Like if I take the banking industry certain aspects, they may be faster in certain things like crypto and all, but maybe the food service and you know, the food and beverage industry may be a bit slower in adoption of those. So it can be multitude of things, but I firmly believe it's a part of digital transformation as well.

Great. So you work for a huge corporation, well known corporation. I assume you've already been through digital transformation, but is it more like a circle? Because you said we're at the age of four, so has it been going, you go through cycles every two years or every so often where it's a constant move to get better and better?

Yes, definitely, because our Cisco is a large organization which has over 71,000 associates and we deliver food to around 500,000 customers across the globe and connected with 90 plus countries with over 350 locations with an annual revenue of around $68 billion. So it's a big entity. And how these kind of transformations work is sometimes it may be a couple of years, sometimes it can be three to four years. But today Cisco is in, I think another transformation where we call it Cisco's recipe for growth. So today we are, we just started this somewhere last year, 2021 and we said we will have a three year plan to kind of grow ahead than the industry growth.

Right.

So we are on it and recent results we shared that we are, we are well above what we defined and we know we are going towards our goal. So I would say in my experience at Cisco it has been like a three year journey. Joe. So we work in three year cycles usually to get to market growth.

Great. So you have a huge title, Director of Software Qual Engineering and Development Platform. So how does that tie into the digital transformation? Do they contact you all the time? Do they see quality as part of the transformation that's needed? Like where do you fit in? What's the roadmap like, what's the strategy look like when you're trying to roll something out? This massive is such a huge company.

Yes Joe, I mean that's a very interesting question. And that's something I wanted to kind of share my thoughts as well and maybe kind of, you know, get to the audience and share some of the learnings as well. That's the whole intent here as well. So if I kind of give you a idea about how Cisco's quality engineering transformation happened in the last three to four years. So we started mostly looking at, you know, kind of a central, kind of a QA unit a few years back and we were kind of focused on more outsourced kind of testing and you know, we were not having cross functional teams in that and we didn't have this agile DevOps movement like this was four or five years ago. Right. And then we kind of took kind of move towards more agile DevOps development I think three to four years ago. And we kind of had a lot of captive engineering added in house, engineers hired and we move into a more from a QA to a QE embedded model. So what it means is basically we move from just talking about quality assurance to more quality engineering. We embedded our quality engineers into our engineering teams. So it made them cross functional and made them self sufficient rather than having to hand off quality into another team. So with that we were going well but what we noticed that happened was this is a massive organization which over I think 50, 60 teams across the globe, we were becoming more siloed. And then we started the testing platforms, the infrastructure for testing the performance engineering platforms and you know, reporting everything started getting siloed again. And we felt that, you know, that might not be scalable model for us and it might, you know, have a lot of, I mean, lacking the economies of scales as well. So what we did was we, we made sure the enterprise testing concept emerged and we made sure that these platforms will be kind of maintained in a kind of central aspect. So our central quality engineering group kind of maintain all these platforms and you know, made sure that the other component teams consume these platforms from a central perspective. And then we moved, we shifted the left a lot we enforced, I mean kind of guided the teams to do a lot of their unit contract, unit integration, functional testing and their performance testing. We moved from just being even a QA selenium, I mean kind of a Java or JavaScript code from a different repo. We made sure that they moved into same repos so that they can run their DevOps tests without having to do it separately. So we moved there and still what happened was when it came to these large digital transformation programs, we were doing well at the component level but when we were integrating these workflows. So when I give a workflow example, it's order to cash. So where we do from our e Commerce or B2B channels and take orders to our ordering systems, going to, you know, our routing and transportation to warehousing and finally delivering that item back to our customer. We had challenges. So we had an opportunity that there was another layer of testing involved in this program or value based testing. So what we did was Joe in last year when we had this new transformation journey, we brought in another layer of testing where we call it more end to end testing, coining digital assurance, where we set up this team that would be end to end, ensuring all the workflows are being tested as well as additional infrastructure team for test infrastructure to manage the infrastructure and make sure that these environments are set up for these testing activities as well as having a core performance engineering team that goes across. And then when we did that, we made sure we had opportunities to see where ever it was possible for us to automate these things and get to a better state. And through that we also looked at another thing like with the digital transformers, the massive thing was test data. Test data management became a very interesting one. We were able to do great at the component level, managing our test data within a SIEM of five to six people. But when it comes to a large engagement and when we were constrained by certain resources, we had to start thinking about test data management differently. So with all these efforts, the enterprise, our leadership, our organization realized the value of testing at a very different level. And since then we've been engaged with all the transformation efforts starting from Omnichannel to E Commerce to personalization to HR finance transformation and basically the entire aspects that need this service. So it's also a test as a service like we don't try to do for each and every project. But these large programs there is a need for this. So I would say digital transformation has been part of QE and QA has been an integral point of it.

At Cisco I mean it's massive initiative and I could just think I used to work for General Electric. And so when they rolled out digital transformation you had real old applications. So it's like how do you even fit this into a digital transformation when it's like still a mainframe actually in the back end. So. And you can't use Selenium for that type of testing. So how are you able to handle that? Like you said, are you just a team that sits outside of these other teams and other ones when they need help, they consume your services or is it forced on all the teams thou shall follow this type of strategy.

Right. So there are three components to that, Joe. So the first thing is like as we move into a digital transformation world, we are extracting certain logic from core legacy systems and moving into a more modernized microservices based architecture. So that's happening as we move. So that way we can invest on good true automation DevOps practices at the component level. But when it comes to the question you asked in terms of qa, I would say that we have a dual approach there. So from an end to end point of view, when it comes to these high level end to end tests, certain things cannot be automated. We are looking at multiple systems across the board. So what we have done is we have made sure that it is at a very high level and touching the end to end flow which is similar to like a UAT test. So this is separate and wherever we could automate, not the end to end flow because it's flaky and obviously we don't encourage, you know, doing test automation at the UI level. We try to do it at the, you know, the lower levels. We make it possible for automation at possible areas. But what we do is additionally we do certain things like test data and test environment provisioning and certain automation aspects there so that it accelerates the overall testing. And to your. The third point I wanted to mention is that what we do at the component level is we would kind of try to enforce and kind of put some governance according to before this comes into the acceptance testing area. Like basically coming to this end to end level, we put a criteria for them to kind of make sure that they have their relevant coverages, their code quality, their unit tests and all that prior to moving here. So we do that, but also we provide our platforms like as example, we provide a Selenium based automation platform, rest assurance based API platform, APM based mobile test automation platform. We provide the CI CD infrastructure if they want to do their execution, we have Separate setup for reporting where they can, you know, look at automation results and do analysis of them. And also AWS and cloud based execution environments so that they can do. So we provide that, give that service to them as well, but also enforce, kind of do a governance to make sure that they do that. Because if they do not certify at the component level, it's going to leak defects into an end to end level and then ideally end to end will leak it to production. So that's how we approach it.

Great. So you mentioned a few challenges and a few ways you've overcame it. First one, you said you had challenges with test data management. I'm pretty sure everyone deals with that. So I'm curious to know a little more about that. Can we unpack it a little bit more? What was the challenge and then how did you solve it?

Yes, so the main challenge came, Joe, Basically it was from our data that we found in production analysis. Right. We found that we were kind of leaking certain defects into production where we had, you know, overlooked certain data elements. Right. So that was the main motivation for us to look at it and try to solve it. And then we did another assessment with some of our partners to understand some of the challenges that we have in our teams. And what we understood is that at least around 75 to 80% of all our teams have test data management, test environment challenges in a given sprint. So that means that they have to delay their testing. The testing becomes a bottleneck. Sometimes overall testing cannot be accelerated. We might, you know, hard code data. Right. And we might, you know, not improve our test coverage because we don't have that. And some also the one of the other bigger areas from infrastructure point of view was our test data was not refreshed most of the time. And even to do certain test environment aspects was taking challenges for us to provision it faster. So handling those were kind of challenging for us. And then it all resulted because we had complexity in our systems, right? We had multiple data sources, you name it, like in terms of different database systems, you know, handling multiple versions of applications and additionally adding to it. We had the legacy environments as well. So these were the challenges. It's a multitude of them, not only from a testing point of view, but from an enterprise scale of doing things as well.

So what did you use to solve it? Was there a solution or was there an approach or guidelines you gave to the teams to help them overcome that or is it baked into the environments? Yes.

So again, Joe, it is a multiple set of activities and I think that's something I want to kind of emphasize and maybe share with our listeners as well. So it comes from the leadership and the cultural aspects, Joe. I mean, I want to emphasize that as well, because if that is not understood and the right investment is not made to kind of solve some of this problem, we are not going anywhere. So, I mean, one thing I want to emphasize as lessons learned is that test data management, test environment management, and don't forget test automation, quality engineering and all that as a whole parcel of our activities. If we do not look at that from a value investment point of view and look at to invest on it and improve its capabilities, we will not go anywhere. So I think the biggest support that we had was leadership, looking at it, understanding that this is going to be a challenge and doing the right investment, starting our digital transformation last year. So that that was the key enabler for it. So that was one aspect. I want to call out that. Because if that is missing, if that funding aspect is missing, and if you don't have the right resources or the tools to do that, we are not going anywhere. Right? And so how we kind of approached it was in multiple. And then it's a journey, Joe. I mean, we are still going through transformation from a digital point of view, but we are addressing it in a kind of iterative, incremental manner. So one area that we started doing the test data engagement was that we had a current problem of test data reservation. So when it comes to engineers, they will, they will look at test data and they will start doing their UI or certain API tests and they realize that, you know, all the data gets corrupted when they all use it together. So the funny thing is like, you know, engineers get together and, you know, keep a shared Excel sheet and keep the customer data all this mapped and, you know, highlight saying, you know, I'm using this. You don't use this in kind of a manual way. So we started looking at that. That was the first pain point, right? So my advice or recommendation to listeners again is look at one or two use cases, right? Look at them and pick the few ones that can be easily resolved. So we started looking at that and we started automating it in terms of how we can get our engineers to now not going into Excel and, you know, do that, get into a kind of a system like a kind of a developer portal and get test data reservation sorted out. So that's something in the work in progress and it's kind of incrementally happening as well. That unblocked a lot of problems that we had initially. So that was one angle. But additionally, what we started looking at was how we can virtualize some of these environments, how we could do some level of segmentation in terms of taking certain gold copies from the production and making sure that we had the right data for our teams without doing costly refreshes in time to time. So what we went about doing was we started looking at certain tools that we could purchase from out and kind of start implementing them in kind of incremental manner. That way we reduced the ability of delaying these refreshes and test environment readiness challenges and got that also solved. And the aspect that we started doing, and this was kind of initially as well, we started looking at Test Pyramid from a holistic point of view, because when we started looking at some of these teams, we, we felt that they were investing a lot of testing on the UI level. So obviously, when you do a lot of testing on the UI level, you are dependent on test data because you need to have that customer kind of experience data coming in as input and capturing it out. So what we started doing was rather than trying to heavily invest on the test data aspect and have less roi, we encourage teams to look at the Test Pyramid and focus more on the UI level, the component, the contract level of testing, and the integration level, rather than getting into the ul. So that kind of helped us eliminate some of these unwanted challenges that we had to kind of improve testing overall. And then additionally, we kind of started, I mean, looking at assessments across the group, talking to teams, looking at their current state and identifying the ecosystem types of db, size of the DBS data types, the legacy modern application mix, and trying to paint a picture of where the problems are and to kind of, you know, address them as well. And then additionally, like, I think the main thing was that with all these aspects, we made sure that the platform kind of approach was adopted. So as I mentioned to you, Joe, earlier, when we transformed qe, we had this platform for our test automation, performance, testing, reporting and all that. So now we have a platform, kind of a team for test data management and test environment management, which is like an enterprise team as well. So we did that investment because the challenges that come day to day, we cannot delegate to a feature team or a delivery team to kind of, you know, look at it has to be central across the board. And Cisco being a global company, we need to look at it from a global point of view as well, because there is a lot of disparate systems across the board. So that was another aspect we looked at it from a TDM TEM point of view, so that it gave that consistency across.

Nice. I love your approach, focusing less on the UI and more on the component level, unit level for sure. Now the environment also. I know back in the day when I worked at ge, the biggest problem was you'd had all these different teams that knew a small part of the installation of this application and so it took forever to run a test. So I assume being at Cisco you have maybe similar issues that using virtualization of environments maybe made it easier to build these environments that was needed since even you're like a third party service, you may not even know have the expertise to set it up. But if someone sets up the environment for you and it's virtualized, it's easier to test. Am I understanding that correctly?

That's correct, Joe. And just adding to that. Right. I think the cloud transformation that Cisco did from, you know, going from, you know, mainframe or like in house systems into the cloud and now it's like a multi cloud, even though we have certain legacy system that is also in a kind of a Equinix cloud. So it's also cloud. We don't have anything on prem. Right. So the cloud journey has enabled us to get there. If not spinning up these environments would have been a nightmare. And the cloud elasticity of quickly ramping down, ramping up has really helped us as well, where we give the engineering teams the ability to kind of help with the central team to kind of do that. Additionally, I think another aspect is also when it comes to testing, testing compatibility, testing on various browsers, testing on certain OSES machines, our testing infrastructure for us to also execute parallelly and in multiple environments and even, you know, multiple combinations and get the tests done in a small number. The cloud has helped us to kind of expedite. Earlier we used to have it again in house where we had a lab and we had the mobiles, we had the machines and servers to kind of do that. But now we are in the cloud and then it's not only from a test environment where it's benefiting, but from our test infrastructure where we do testing. The cloud has accelerated that as well. So it's a dual approach. So it's very important that a testing team looks at that and starts investing on those things as well.

Yeah, I can't imagine anyone nowadays not having that in the cloud. Just back in the day, once again we had a lab in the lab had like different versions of Windows and you had a test on each one. It was crazy. So it's awesome to see that that's actually going on. So I guess you know what's next. What are some challenges then that you think could be solved in your next iteration of the transformation? I know there's a lot going on with AI and machine learning. Are you investigating any of that or is that more too early for what you've been focusing in on?

Yeah, I would call out two things Joe. I would say one is the AI M aspects and this is mostly related to low code, no code applications like I mentioned about the more end to end testing and also our UAT folks, that is our business and operation folks across the US and the globe doing that. So what we have now come to in place now since we established this strategy for the last one one and a half years on this end to end digital assurance journey, we feel that there is an opportunity for us to kind of bring in low code, no code or more kind of AI ML based solutions to do software testing at that level because we don't need to Invest on Java, JavaScript kind of, you know, knowledge because let our component teams have that rigor, you know, the cross functionality and that full stack QE approach because I think that needs to be there for a solid quality engineer. But when it comes to large programs, if we are taking couple of weeks to do certain things at that end to end level or U8 level, I don't think it's needed if we can quickly get them to script and do scenarios on certain of those tools. So that's something we are investing on looking at it from a kind of business case point of view, the value trying to put the numbers across and see if we can really materialize it. The only challenge that we might have is how certain interaction works in terms of technologies that we have. But that's something I think I have seen that it has overcome. The second aspect is that what we are doing is like in terms of data testing event driven architectures because we are heavily focused on Kafka based event driven programming. So we are looking at those aspects because that is going to be the next level of our testing in terms of maturing. So we have already built certain data testing frameworks with it as well using Python, Pyspark and all that. So that's where we are moving towards and want to kind of invest more on it. So those are some of the new areas we are working on it. Other than that I think going into traditional, the UI based, API based automation where we can use for hailing and all that. Those are Things that we are investigating as well.

Yeah, I love that approach. I think that a lot of companies miss out on. Now there's applications in HR and financial apps that you normally wouldn't test, but because it's usually Salesforce or some sort of package based application, but using low code, no code probably would help with that. And I love how you're not forcing a tool on like this is the platform thou shall use. So I guess that brings up an interesting point. A lot of times certain teams have certain preferences for tools. Like someone might like Cypress, someone might like Playwright, someone might like Selenium and they might be more successful one tool over the other. Does your platform team force them to use your platform or do they have the lead way to say, look, we want to use Cypress for this?

Right. So that's a very good question. And it has been evolution, I think with my experience for the last, I think 15 years, working with previously with, I mean Macy's.com Kohl's and Cake Lab, coiscoLabs like it has been evolution. So I have seen ups and downs and it's very interesting to see. And I think Joe, going back to your, some of your past podcasts five, six years ago, I mean we have heard many, many concepts, right? So what my belief is, Joe, like we should let the development team, you know, have that autonomy, right? At the end of the day they should have the tools that they prefer to do it in a faster approach and we should kind of help them to kind of, you know, use that tool. However, when it comes to an enterprise at scale, you know, at scale as big as Cisco, that's why it's very contextual. My answer, because if you ask this from a more of a startup organization base, I would say go with the easiest to kind of get the value sooner, right? But when it comes to large scale organization, if I have a hundred teams and if all my teams are doing their own level of testing across the board, there is a velocity problem that usually would create, right? So if they, let's say we are using Cypress and they are investing 10% of their springtime to learn Cypress, right? So whereas we might have a kind of a good Java or Selenium based tool or maybe a JavaScript selenium based tool platform already built that they could quickly onboard. So there is a catch to that. So I would recommend teams to kind of look at that. But I would want the team, I mean especially if we are looking at it from a enterprise point of view, I would want the team to kind of start looking at it from their developer point of view. So my concept here is like if the development team is using JavaScript and they are in GitHub and in their repo, I would encourage the quality engineering team also to be on the JavaScript and get into a framework that basically, you know, makes sense for them. Right. And then their unit integration and functional testing as a component can be certified and also they should try to, you know, certify their performance within that particular aspect as well. So I would say get into that concept initially and then if there is end to end testing or certain performance testing you need to do outside of that, maybe make sure that, you know, you adopt to a kind of Enterprise Scale 1. However, it's a very, I would say very tricky based on the context and I would encourage team to see if you can utilize much. And then in terms of our platform team, Joe, so we support that. So we have the core utils that can, you know, do everything but we Support for Java, JavaScript and Selenium, Cypress and even performance testing. So we can go from things like, you know, Load Runner, we can go for things like JMeter and even go for things like K6 which is like more developer friendly. Right. So you need to give that ability. I mean it was I think three, four years ago if you asked me this question, I would have said no, I will enforce. But now it's so diverse. You would, what you should do is like keep your platform like your reporting, maybe execution kind of consolidated because you don't want to duplicate reporting and everything and give that test framework and you should be able to plug and play, you know, Cypress and you know, JavaScript tool or Java tool so that you give the flexibility for your organization to kind of do what is best for their velocity and feature development.

So I assume you designed the platform in mind with an interface or an API that could be consumed by these other tools so that ultimately you have the same dashboard, the same metrics being recorded in one place, regardless what the heck they're using. So I guess that would be important.

Exactly. So when you do architect and build that solution, I think you need to start looking at that and keep that abstracted so that you can design and develop for it.

Before we go, is there one piece of actionable advice you can give to someone to help them with their automation testing efforts or digital transformations and what's the best way to find or contact you?

Sure, Joe. So I think my recommendation in terms of looking at large enterprises and looking at digital transformation is that I Think it's important that as leaders, as directors, as managers, kind of emphasize the value of this to your business, to your leadership, to your cio, to even your CEO. Because at the end of the day, the digital transformation without quality or that business assurance is not a digital transformation. Because if you are leaking things into production and your customer experience is suffering and your stability and your performance is suffering, then that digital transformation is not a true digital transformation because it's all about customer experience. Right? So I would encourage, I mean, I'm not going into more technical aspects because I think there is a lot of things out there. But my advice, based on lessons learned, is that encourage your leadership to kind of invest on it. Put a business case, show roi, show the value that you can get out of it, and you know, work towards a plan to kind of invest on some of this. Because at a large organization, if you do not do that, investment cannot reap the benefits of your digital transformation as well. So that's number one. And then number two is I think you need to build a solid team focused on different aspects. So you need to focus on a team that is focused on strategy, more on test automation, performance, AI ML. So you need to have a strategy team, right? You cannot get your strategy team to be on the delivery, right? And you need to divide that with your delivery team, you know, your QA engineers in your teams cross functional or your end to end. So you keep those two together, don't keep those two together. You keep the strategy team and your delivery team separate and kind of influence them to talk to each other and, you know, create that competition for them as well. That will really help you to kind of build a team. Because if you don't do that, you will, you know, get into so much of challenges with delivery. You know, you may have production outages and all. You will miss that. You know, you will miss the innovation, you will miss that kind of aspects that the strategy can do. That's two advices that I have. And Joe, you can, I mean anybody can contact me on LinkedIn or I'm on Twitter as well under Dilhan Manavadu. So love to connect and you know, answer any questions that any person has.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode, head on over to test guild.com forward/A4 1, 6. And while you're there, make sure to click on the Try for free today link under the exclusive sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.