Shift-left DevOps Testing with Vishnu Nair and Geetanjali Gallewale

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast and my goal is to help you create DevOps toolchain awesomeness. Hey, it's Joe and welcome to another episode of the Test Skill DevOps Toolchain podcast. Today we'll be talking all about shift left DevOps testing. If you don't know, Vishnu is a QA advocate currently working at as a QA architect for a very large bank and he has over a decade of experience in multiple things like designing, curating and writing test cases across multiple technologies including big data, UI and backend services. He also has expertise in performance engineering, resilience testing and security testing. And he has a lot of experience LDQA tools and frameworks across different functional domains and technical stacks. So really excited to have him back on the show. Joining him we have Gitanjuli, who is a seasoned technology lead and has a lot of experience with development and automation testing, has a wealth of experience in driving strategic change and organizational transformation, executing complex client engagements, and delivering enterprise scale test automation programs across leading global banks and IT organizations. And really excited to have them both on the show because they both have hands on experience as well implementing DevOps and DevOps testing with a really complex enterprise company. So really, if you want to learn real world experience on how to really implement DevOps and how to really up your testing in this era, you don't want to miss this episode.

Check it out.

Hey, if your app is slow, it could be worse than an error. It could be frustrating and in my experience, frustrated users don't last long. But since slow performance is a sudden, it's hard for standard error monitoring tools to catch. That's why bugsnag, one of the best production visibility solutions in the industry, has a way to automatically watch for these issues. Real User Monitoring it detects and reports real user performance data in real time so you can quickly identify lags plus get the context of where the lags are and how to fix them. Don't rely on frustrated user feedback. Find out for yourself. Go to Bugsnag.com and click on the free trial button. No credit card required. Support the show and check them out. Hey Vishnu and Gitanjali, welcome to the Guild.

Hey. Hi. Hi Joe. Thanks.

Great to have you. I really love people that have a real experience implementing automation and DevOps in the real world. So really excited to dive into this Topic I guess before we get into it though, people may have heard of shift left testing but maybe not necessarily shift left DevOps testing is it the same? Is it different? How would you explain what is shift left DevOps testing?

So if you see Joe, shift left is a wide concept. If you can see across the industry shift left is used for the different nomenclature. If you go with the auto industry or the everyone talks about ship like but whenever it comes to the IT industry the ship left always talks about the shifting left to the testing. But whenever we are talking every then it is correlated with the shipment with the testing and when it go with the DevOps or Agile methodology as well. These are really a high performing teams required this as a guideline and principle. But still there are challenges to perform. And at that time every time shift come as a we thought we can say as a savior for them that saying that ship left is a good and let's move the ship left with the testing. So yes, most of the time we say yes ship left is related to the testing but I would say it's not only testing but when it is utilizing properly with the DevOps and Agile then we can say yes, shift left is a good for agile and DevOps testing is a one part of it.

I would say absolutely. I have this debate all the time. A lot of people say testing has nothing to do with DevOps but I think it clearly does. I think a lot of people just focus on functional automation as testing and they forget about all these other types of automation activities and testing activities. So maybe can you give us a few examples of other types of like DevOps more related activities that you can shift left and then we can dive into each one of those.

Yeah, so when we talks about DevOps or in the Agile most of the time everyone talks about the feedback loop where we go with the continuously feedback if you are gate and how you will get the feedback continuous feedback. So most of the time it gets correlated with the continuous testing will give the feedback and for that every time it talks about the automation is there then it will be a feedback loop. It is more and when it comes to the DevOps or Agile when it goes to the faster delivery at that time. So it talks about only on the functional testing part of it. But I we here we are talking about when it goes with the shiplet it's not only with the functional test but we have to go beyond that functional testing and have the shift left not only for functional testing but integration testing, regression and some basic part of the Performance for the stability purpose when it goes to the production.

So basically what exactly? In our project the main idea with the shift left approach was that to shift everything, the testing part as well as the DevOps part to the left side of the development. So to go hand in hand actually. So we are actually approaching for a V type model actually where the testing is happening along with the requirement analysis itself. So if we talk about the Agile principles and we follow the Agile ceremonies. So as soon as the PI plannings are happening the testing should, you know, the mentality of that testing should be also in place with that along with that. So the requirement analysis in that time, the testing, you know, the tester should be. It's not about the just only testers but everyone's mindset should be also in focus of the quality. So everyone should be aware of what the testing is happening and what the main focus should be on the quality of the product as well as with the development.

So you know, I find it very difficult. Well I did. Maybe it was the company I worked for. Difficult getting everyone buy into testing in general. How do you get people to embrace this DevOps then where it's more and more includes developers and everyone else on the team. How do you get your culture or your team really buy into this whole shift left type approach?

Yeah, so it's not an easy job because it's needs the mindset change as well. And when we are talking about the mindset change, not only mindset change but there is a skill gap as well when you are talking about the shift plate and with the Agile team everyone is a team member. We don't have these silos like QA or dev. Everyone should start working on it. So there are different challenges as well when you start bringing these concepts inside the team. So we also faced these similar challenges when we started with this shiplap approach in rt. Basically challenge number one is the mindset where initially even though we have a DevOps and agile, the team as a small sprint is working as a small, we can say as a not fully sprint team but as a mini agile soft model is followed like a SDLC model where development is done. After that tester is picking up that story for the testing and then going. So as Vishnu mentioned initially we asked team to start from the requirement page itself. So everyone is onboarded. Now second challenge where we are having to break the mindset. So for that breaking the mindset we need to onboard the QA schemes on each and every person. So we started with the knowledge sessions where how QA's mindset can be developed and how we can have the QA mindset and developers instead of developing the code, how they can think from breaking the code itself. So that is the area where we work. Apart from that, we work on the technical side where Vishnu can give the more idea about it, how we can involve tester, the developer in a testing using the programming itself.

So yeah, so main the functional part was, you know, train the developers from the changing of their mindsets and everyone on the team to have a QA mindset so that they can break their code. So this was the functional parts from the technical part. What we incorporated is that we used a certain kind of framework which you know, was more focusing rather than writing the code to break the, you know, but to write right, good test codes actually. So we were used earlier, we had some, you know, this project is elaborated so we have big data, etl, backend, ui, everything is in place actually and it's a huge background of the project. So very big team is working and there are a lot of QA's also involved in this. So we needed a framework which, you know, we can design and the testers who are testing, when I'm saying the testers, it's an unique term. Anyone can test it actually the dev team or the testing. So their main focus should be on writing the testing test cases actually rather than, you know, depending on any framework itself or writing any code on the framework side, they should be only writing the test case so that we have a higher productivity on the feedback loop. So for this what we did is that we used Karate framework, the DSL where we are, you know, writing only the keywords so that we can write the test codes. So we have this same framework being used for ui, API, backend, ETL as well as for big data automation as well. And since we only have to take care of the keywords and if we place it correctly, the test code itself gets developed. So in that fashion what we did is that we used the framework to have a faster automation script generated for this. So in this way we were able to attain, you know, writing the test case on the pre development side itself. So as soon as the story is in place and developers are working on the development side and the tester who, whom were maybe testing on that, they can write the test cases parallelly on that. So rather than they are focusing on the framework and how the execution will happen, they are now focusing on writing on the test scripts only.

Yeah, and add on it, that framework is such a way that no one has to learn for it and there is a very less learning curve. So that is other process change. We applied that. If anyone wants to onboard on it, he or she don't have to give a more learning curve. He can start immediately.

No, it's a good point. You know, especially when you're dealing with developers, you're like, oh, the developers, they should be able to pick up Selenium or something and code it. And in my experience that is not the case. So even using a keyword driven framework, a lot of people think it was just for business users, but it sounds like it's also a driver for developers and testers. So you're speaking the same language and not getting caught up in the like the technical geekery of testing almost.

Right, so the karate DSL actually helped us here, actually. So we are using the keywords and so that they can understand what exactly is happening with the. So whoever is reading, you know, the test code is aware what exactly is happening with the testing. So everyone is involved from the dev team, from the stakeholders, from the project managers, everyone is involved in the testing side and they are exactly knowing what that on because we are following the acceptance criteria. So based on that acceptance criteria we are writing the test case. So everyone knows what exactly is being developed and what exactly is being tested.

All right, so like I said, you work for a huge band, big, big organization. How big is this group? Like do you have to have like an architect on top? Like do you actually lead the effort with each of the teams? Do you have to know who has training, what they need to know? Like how does it work as a part of like you have a new hire and the new hire goes through like just how does it work? How do you get everyone involved and how big's the team and how do you make sure everyone's on board and using all the right processes that you have in place?

Yes, ours is a big team. We have around 100 plus teams, team members involved for our project where we work for the surveillance. When we are talking about surveillance, it's a huge piece data we are getting which needs to be processed and alert and after getting alert report it to the business for the further actions on it. So we have enterprise system as well. And as you mentioned, yes, we have around six spring team working on a for each features, different features on top of it we have architecture layer on top of it, we have the business layer on top of it, we have the management layer as well. And when I'm saying about this business Layer or the architect level, it's across node because surveillance for being a big bank presence across the world. We have the branch data collecting from across the world. So data is also huge. So when we are talking about data, it's unstructured data where it is coming from the bank, whether in a different type of languages as well. Because when we are talking the surveillance for the data which is going. So it's all type of data, audio, video records as well as email or chat or social networks data and even for the regulatory system. So all type of data coming. So if you can see for us, backend means ETL jobs as well as database as well as the Hadoop where everything has been processed and structured and then it comes to the ui. So if you consider our presence of the project guideline, 70 to 80% towards the backend and just 20 to 30% on the UI side and we need the framework where everything is there. And that's where the Vishnu had clearly mentioned that when we are talking about QA skill set, one person can't be everywhere that the backend that's thing as well as ETR as well. And same with the development team as well. So we need to onboard it as well on the every side. And that's where. Yes, as you mentioned for the training perspective. Yes whenever there is a new onboarding person come, new person come, we have to give the idea about everything.

So to add on that we are taking regular sessions on that actually monthly we have one session throughout the entire team actually. So whoever is new joining, we make sure that they are joining and we also have a feedback loop. So earlier we actually make some, you know, who have worked previously on the shift left, we invite them and we generally provide a, you know, a feedback how their experience with the shift left approach was so that, you know, it builds a confidence on the new onboarding members, how well it is right now working and how well they can contribute to that.

I guess I don't know why I'm fascinated by this but once again I'm going back to our banking company. So a lot of times when people think agile and shift left and like all the latest and greatest, it seems like they have to be more like Google or something where you have like maybe newer technology. But I used to work for an insurance company, I used to work for healthcare company and they have older technology and people usually say oh we can't use, we can't do CICD, we can't do DevOps because we have all this legacy software or we have all this regulation. So working for a banking company with you probably have all regulations as well. You have to work around. ETL is not simple. You're not doing simple browser based front end type automation. So how did you do? You didn't have any challenge to overcome in that aspect where people are saying like well wait, wait a minute, we can't move this fast or we can't shift left because we have to follow these different processes because we're regulated or something or we have to worry about being audited or anything like that.

Right.

We had these challenges actually since we are having audits as well and we have certain limitations on adapting a new kind of framework or new type of any open source. Because being on the regulatory side we have to make sure whatever we are using are stable on the stability versions and they are secure actually. So yes we had this initial challenges. So how did we overcome is that we actually have a very good DevOps team and the architect team actually and we ourselves has go back through the different back to where we have started earlier. We actually go back to whatever challenges we have and what all are the solutions that are available in the bank already and using that we accommodate, you know, few changes we did ourselves. DevOps teams helped us in setting up the CICD pipelines and still they are, you know, providing the support on whatever friends they need. So either it is, you know, any Unix side or any, it's a database side. So definitely the DevOps teams and the architect team are providing the better feedback and help to us so that we are able to achieve.

So you both mentioned feedback multiple times. You know people chuck shift left a lot of times they don't realize that when you shift left you're, you're getting information from production after you release and then building that back into the system from the beginning. So how do you measure then how you're doing? Like do you have any metrics or KPIs used to to realize okay, we did this release went successful, it didn't go so well. Let's learn from it and then build it in. Now when we start from requirements again with our next Sprint or next feature.

Initially what you mentioned, yes, we used to do that but now we have the control environments available for us for validation. And when you talk about the control environment we get prod like structure on the control environment. It's a prod like replica where we do the feedback loop and where we get the production grade data. Production grade manufacture data where we can accommodate large volume as well and we can do the performance and continuous feedback. So from the regulatory perspective or the what you are talking about, yes, there are still restrictions, but with the controlled environment we can have the feedback loops available.

All right, so how do you create then a realistic control environment? Because once again when I was in healthcare it's like, you know, we ship to these different hospitals, their environment's completely different or it was too expensive. What are you using to build those environments to make sure that they're accurate.

So it's a replica of production and it will be like a production environment only with restricted access. So when we are talking about these, all these big Google or Amazon, they have their replica where everyone can access including dev team and they can use it. But in our case it's a control environment where limited users have the access from the architect international small team who can execute and replicate it. So in integration environment everyone can do and replicate it is again the similar structure. Like from the budget perspective, yes, we get the budget where we can replicate it and have the production like environment. But it will be with the restricted access and with the limitations. But it feels like a prod. So broad replica, you can say we.

To add on that we have a particular team for that actually who are doing unmasking all that stuff so that you know, the production like data can be replicated. So we are not entirely keeping all the production data. It is just you know, a small portion of the production like data which we can mimic for our environment and for which we can run our automation test cases on that actually and can provide the better feedback.

Because that was my follow up. Like the data, especially in banking, how do you get realistic data that is like production but you can't use it real customer data obviously because you're going to be regulated and you're a bank masking. But is there any more that you can give how that is done?

So we have the separate teams, CDO teams available to us is the central data organization which provides us the data on the all the environments with the manufacturing whenever it is required for the lower environment and with masking when it comes to the controlled environment.

Very cool. So now you talked about karate. Is there any other tools or things that help help to enable you to shift left with DevOps that you find helpful?

Karate is the main tool that we are using for the automation but for performance testing, right. We have a huge data and we have to do some of the infrastructure testing and as well as some of the performance testing on the API sites for which the UI is taking care of. So for performance testing we are definitely relying on the JMeter where we are writing the custom jobs for that actually for the infrastructure validation. So infrastructure validation is not a regular basis. We are doing it is based on the infrastructure changes, if any we have or any upgradation we are doing on the infrastructure then we are running the performance script on the infrastructure side. For the UI and API side we have Load runner. We are using the load runner to run to execute and mimic our user experience. And for at the Sprint level also we are trying to execute our performance scripts using the Karate itself actually. So. So we are writing the scripts in the Scala format and Gatling, the inbuilt mechanism of the Karate is helping us to test the performance script within the Sprint timeframe.

So who handles the performance and resilience testing then? Is that just testers or do you also expect your developers to contribute or is it just the whole team? It's just part of a feature. Okay. In the Sprint these are the definitions are done, doesn't matter who does it, but it needs to get done right.

So we are actually doing the initial phase when we were starting right before the shift left, the testers were only that part of the team. But now right now since we are moving with shift left. So anyone who has, you know, done the scripting because once the feature has been designed by the Karate, you can just call that feature in the Scala like language and then you can just run the performance step. So anyone who is, you know, working on that feature, whether it is a dev member or any QA member, they can execute the test cases for the performance.

All right, so if someone's listening to this like this sounds great. I work for a banking organization, I want to do this as well. Any caution you would give them or any like advice on how maybe to make it maybe a smoother transition than maybe some things that you may had to have dealt with or you dealt with when you were first starting to roll this out yourself.

Yeah, so couple of things I would like to give the benefits that one thing. Whenever you are starting from scratch, don't work as a silos where the developer will do a development and Q and a build a T shaped team where at least some knowledge about the testing or about the architecture developer and so on. And when it comes to the tester it should have a development background as well or he or she should also be able to contribute on the dev side as well whenever required or at least from the analysis side as well. So T shap team is the must to have when it goes with the shipler, that will be the keyword. And second thing, go for a framework automation feedback loop which can be get utilized from functional testing till performance test without any maximized change. So whatever is required for the changes from moving from functional testing to integration testing, it should be just plug and play from the functional test script. So that will be the give a good advantage for the ship. And third, and the last thing, don't focus on only build automation. Whenever I talk about continuous testing is also must to have the ship play. And that feedback loop from the continuous testing is must to have. So from scratch when they are developing, then these three points should be looked into it. Right?

So my giveaway would be right, the performance should also be included in the testing mainly from the infrastructure side. Because infrastructure is a costly environment and we are investing so much and once we have everything on that and at the last we decide that this infrastructure is not supporting the data what we were supposed to have. So infrastructure testing should also be a key portion when we are having an infrastructure being designed and it should be tested as well.

So how do you improve the system? So you had a new tool or technique like AI is a big rage. Do you research different technology like and say okay, let's try to incorporate this now and see how it can help us. Like how do you continually improve their process to make sure it's continually getting better as well?

Right. So we are analyzing on the market what we have in the markets and we are trying to implement the current trend with the current trend. So AI is a big player right now that we have and it definitely has something to provide on the testing as well. So as a team also we analyze what can be improved on the AI side. And as an individual contributor we definitely go back to the market and we try to have a continued conversation with other team members who are working in different organization and how they are working with the AI tools if they are using anything. And we try to get a feedback from them how well it is so that we, if we come back to our organization and we see there is a chance that we can implement these changes, we can incorporate them.

All right. So one other thing I'm dying to know is obviously you need money to do these types of initiatives. How do you convince your like your C suite, your CEO, like your executives, like hey yeah, we're going to do this and this is why it makes sense. Is it coming from them to tell you hey, we need a better DevOps process or is it coming from you all saying hey, this is what we're going to do. And and then you have to get buy in from your team, your executives to do it.

I would say it's both way. We have in a bank hackathons going on even on the innovations going on every year and the competitions going on their innovations. We can put forth our ideas and the best idea gets selected and get budget as well to implement it. So and even in a hackathon we gathered the ideas and best ideas we circulate across the team to develop it and whatever has been developed can be utilized and worked on it as well as it can be a product as well for a bank.

Oh, very cool.

These things in play.

Nice, nice. All right, before we go, is there one piece of actionable advice you can give to someone to help them with their DevOps efforts and then what's the best way to find or contact either one of you?

So I would say if you want to go with the jobs with the one good advice, start with the B model where you can have TDD and PDD approach combinedly work where TDD can go with the unit testing and the PDD can go hand in hand with the TDD for acceptance test. So have a framework with that so that you can achieve shift better.

So from my advice would be, you know, rather than focusing on the manual testing, the effort mostly should be have on the automation side as well because automation even though the ROI on the automation side for the initial phase is less but as soon as we grow, as soon as our project grows, the ROI seems to be have on a higher side on the automation side. So it's better to have the automation working parallelly manual. In some cases it's beneficial, but it's always to have an automation scripts to have the, you know, pure benefit you can get.

And for links of everything of value we've covered in this DevOps toolchain show, head on over to test guild.com P126. Remember, latency is the silent killer of your app. Don't rely on frustrated user feedback. You can know exactly what's happening and how to fix it with Bugsnag from SmartBeer. See it for yourself. Go to Bugsnag.com and click the free trial button. No credit card required. So that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness as always. Test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world world. We'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.