Change Controlled Continuous Testing Pipeline on Cloud with Parasar Saha

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast and my goal is to help you create DevOps toolchain awesomeness. Hey, it's Joe and welcome to another episode of the Test Guild DevOps toolchain. Today we'll be talking with Parasar all about change controlled continuous testing pipelines in the cloud or on the cloud, if you don't know. Powerstar is the founder and CEO of Digi4, home of the Cloud first testing solution built on an ecosystem of cloud based products in qe. So really excited to talk to him about this. He has over 21 years of experience in quality engineering and extensive enterprise Automation knowledge and DevOps knowledge as well. Prior to founding his company, he was the head of quality engineering and global organizations like Air Canada and other large corporations. He's been a global evangelist and speaker on QA Ops DevOps for a while and he also organized or is the organizer of the Toronto Testing Meetup, which I believe is I think the second largest testing meetup community in Canada. So really excited to have him on the show. You don't want to miss it. Check it out. This episode is brought to you by SmartBear. As businesses demand more and more from software, the jobs of development teams get gets hotter and harder. They're expected to deliver fast and flawlessly, but too often they're missing the vital context to connect each step of the process. That's how SmartBear helps. Wherever you are in the testing lifecycle, they have a tool to give you a new level of visibility and automation so you can unleash your development team's full awesomeness. They offer free trials for all the tools, no credit cards are required, and even back it up with a responsive award winning support team show on your path to great software. Learn more@smartbear.com today. Hey Parasan, welcome to the Guild.

Thank you Joe, like you know, glad to be here.

Awesome to have you. I know you've joined us on the Automation Guild Automation podcast. This is the first time on the DevOps tool chain. So really excited to get your point of view on DevOps and how QA fits into it. So I guess the first thing is anything I missed in your bio that you want the Guild to know more about? I usually butcher the bio, so any. Anything you wanted to add?

No, I think you actually covered like, you know, things like one Thing I wanted to mention, like I was very excited to come to this show. Like I have been two of these shows before, right? And after this testing cloud show, like a lot of people kind of watched it and when I meet them, like they mentioned like, you know, oh, I watched you in the testing cloud, right, you know, that thing. So I felt this is a kind of a natural progression going into that like where we will talk more how you can take that testing cloud into implementation mode. So I was very excited to be here.

Awesome, Greg, great, awesome to hear feedback like that. So you know, the topic is change controlled continuous testing pipelines on the cloud is what you recommended. So why this particular area that you want to focus in on, you know.

Before even we go to this question of. I want to kind of highlight one point in the industry level thing, right? Is like the world of development has moved on to DevOps. People realize that, okay, only developing code is good, not good enough to bring the efficiency in the software delivery cycle. It needs an operational model called DevOps Now. But there is no mirror operational model like QA ops in the QA domain. So. So what happens is like you build the tools, okay, and you build automation, but because of a lack of a operational model, like it's always struggling to get that efficiency. So if you take this whole thing like there are four quadrant, one is tool, one is data infrastructure, then you have another quadrant as like AI and then another quadrant as reporting. But what brings everything together is the pipeline and that is the operational model, that's the DevOps. And I'm so glad that you are kind of bringing this up on the DevOps as a separate topic. Right, we exactly need to talk about what is qaops and need to bring these QA ops in the industry. Now let me get to your question. The question is like, you know, why to talk about this kind of the impact based kind of like DevOps thing right now when you are using continuous testing like automation is pretty dumb if you don't have a intelligence built on it, right? Let's say you are set up a continuous testing pipeline but if you don't have an intelligence built on it, that which knows when a code is changed somewhere, what set of tests to run and build up a pipeline in a way which is to that extent scalable of what the change has been because it's a big change needs a big amount of like execution capability. A small change needs a small amount of execution and you need to focus on that exact change rather than kind of like running everything Every time you're going to lose lot of money in that space. Right. So that's why it's very important to think about a impact control execution model. And if I can take a moment to kind of bring up a slide. Okay. Which can actually show you that piece. Right. The whole impact oriented thing.

And for the folks that are listening to the audio only podcast, definitely I'll have this embedded in the show notes as well with the YouTube version with the video to follow along if you're missing out on anything.

Thank you, Joe, for that note. Let me just show you that stuff like this, right? See when your code is kind of making entry into your testing environment, like testing environment, need to identify the pulse. Right? Okay. What is change? And then has to use a test management as a control center. And that control center should be able to kind of know what the change has been and pick up the test automation test or the test what you need to run on that. And then it need to pick up a infrastructure of the size of the change or the execution that you want to do. And end of the day you want to report it based on the change that has happened. Right. So end of way, like from the beginning to the end, you can have a change and a control testing pipeline which is exactly focused on that chain. And you can do it by building tags and bringing an intelligent mechanism into this. So this is why I think like it's why we need something like impact control kind of testing pipeline.

No, I 100% agree. A lot of times people say, well, we have the infrastructure and containers, we could just scale up and run every test. But it's very inefficient to me and it causes a lot of noise and a lot of maintenance when you just run every test for every check in. But being able to do that mapping is sometimes difficult. Even teams, if they're using tagging, don't necessarily know that code being checked in may impact test one and five. So how do you make that connection?

So the connections can be made like because those pulses are the chin, the indicators are already there in your commit, right? So commit most of the time you have that information like what has changed? Literally like that will say a story, this particular story has changed. Or it may say like this particular module has changed or it may also have that, okay, this is a large change, right? So once you have that information in your code commit, right? And then you have a similar kind of mechanism on the your test management system. And we got to use test management more as our control system, our Control center, rather than just becoming a still repository of test cases. Right. So here in the test management, you got to have some mirror text. So when you run a test, okay, which test is related to which story, what module, what complexity, what is this test? A part of the regression, a small regression or big regression. From there like you can make these two ends joined. And that's what we also built. Like we build a intelligent module who can read up what the code changes are happening and then kind of like also tag this with the changes which are there in the test that are there for testing those things and we can match them together and build that.

Awesome. Now I think I may have jumped ahead. I think you, I could be wrong. Did you call this, what do you call this? Do you call this testing pipeline as a package? Like what is this kind of framework or kind of flow?

Yeah, so that's very interesting. Like you know, you asked like the package part of it, right. If you look at the software industry, right, Today's world, everybody is becoming a package, right? You know, there's a package software like SAP, there's a package software like Salesforce and thing, right? So but in testing, how are we doing it? Like, we are building frameworks, we are building pipeline. Okay, so we are building tools. But how can we keep up with this? If you think about it, when the development world is using a package and they're making a already a head start, you can't start from grounds up. So that's why instead of relying on building tools and things, like we have to think of taking a product and taking that pipeline as a product and then implementing as a package. Now how we implement that is like we build a model of like adapters and mechanisms of like a processing engine so that when there is a new tool like let's say you are using for like CICD, like GitHub, you can have an adapter of GitHub coming up and it will configure to GitHub. But somebody is using Jenkins, can use the like adapter of Jenkins and do that. This is the same model Jenkins uses that the plugins in the stuff, like the Jenkins plugins and kind of adapt to several things. So by that you can build up a pipeline as a package and deploy the package. Instead of losing time to build it, you build it from a package and start focusing on your test cases.

Nice. So the package, we already know the components that need to be in that environment when you run those tests without you having to configure every time.

Is that what I'm hearing exactly right.

Nice. So how do you achieve then this change control pipeline is it just tags as well that determine what test to run.

It's actually like, you know, different mechanism like and it has a maturity model. Right. Initially you start with very simple like getting up the tags and stuff. Right. Like you get the tag, then you kind of like map it to your test cases and do that. And as you mature in the organization, you start looking at shift right as well. Right. You start getting indications from your production as well. The production may have a particular change which has caused problem multiple time in production. And that may be something which may actually say this is a high priority test. And this may, if you ran that previously, you may already gather that I have run this many test cases along with it. Right. So you do a shift left where you kind of like figure out that okay, what is the test cases to be associated with a change of code. And you do a shift right where you also bring parameters from the production saying that if this thing has happened before, these are the tests I have run. With these two diff knowledge, you bring up a dynamic way of test selection.

Very cool. So, so is this a solution that you have with your company?

Yes, that's something that we are pioneering in the industry. Like you can't keep like running automation test as a static regression. You got to kind of think about how to make it dynamic, how to make it dynamic, how to make it continuous. So you can scale, but you can scale optimally.

Absolutely. Yeah. I love this for sure. Especially as more and more code is being created nowadays, especially using AI. I see you need to be a little more selective with the tests that you're running and using tools like this. So over time does your tool become like you said, it knows over time then that you checked in this before. Here are the tests you ran. So it knows that maybe next time you check it in, let's run these tests as well again.

Yeah, so that's something the intelligence like you know that comes with the maturity. Right. Once the customers start using it like you kind of like start building up from there.

Now you mentioned maturity a lot. When you speak with DevOps in general, do most people have this conversation before they like, you shouldn't confuse testing and QA with DevOps. It's totally separate. You're confusing things when you add the two. But I think they're essential, they're one in the same. Just your thoughts on this. Have you ever gotten conversations like that or why convincing people that are maybe DevOps engineers why QA and testing are essential for DevOps.

I think that's a very good area to kind of talk about. Like, you know, I always get into this conversation with that. I see there is a need of segregating this segment and saying this is qaops because that segment needs to mature. And the segment will only mature when you recognize that as a separate segment. So I tell you this, like, when you see a profile of a DevOps engineer, will a DevOps engineer ever write, okay. That he or she had experience on like Selenium or Cypress? Right. They won't write that. They will say like, you know, I have worked in Kubernetes, I have worked in this. So their profiles are not aligned to what is required for a testing pipeline creation. Right. And most of the SD are looking at. They are not looking at that. Okay. They are Kubernetes expert or they are cloud expert, they are testing experts. So there's a gap in the industry. Right. The people who had the. Both the knowledge, like how do you like Kubernetes, how do you do the pipeline creation and how do you do like Dockers and those things and also have the knowledge, how do you use test data? How do you use like different tools like Selenium, like browser stack, like Cypress, like, you know, different other things. Right. And all together bring up a pipeline. They don't need to be that testers that sting like they are not trying to break the test, but they are trying to build the infrastructure which required to for the test to run fast.

So with this gap then do people need to upskill? Do they use a solution like yours that helps them with QA ops and then almost bakes in that type of knowledge rather than having to expect a DevOps engineer testing or a tester to know Kubernetes. Yeah.

So I think we need to have people who had that niche knowledge or who had the passion, like who are already architects or kind of senior estates. Right. There are so many people in this industry who can actually get to that level of becoming the QA ops. Right. Because these days it's not only tool, it's also infrastructure. Right. So how do you tie infrastructure with tool and build a pipeline? That's where the solution of efficiency comes in.

Very nice. So can you talk a little bit more about that? Then your solution then does build that bridge between infrastructure.

Yes.

And the test.

Yeah. So what we focus on is called two things which are like, you know, not focused much in the industry, cloud in the, you know, I'm not. When I say industry, I mean like QE side, like cloud and QA ops. Right. That's why we build cloud based QA ops, right? The whole pipeline. How can I make that pipeline on the cloud? And how can we take the industry more to the cloud? They create the test, put it in the pipeline, get it in the cloud, move it fast.

So it's like a self service solution then as well. A lot of companies I speak to now have rather than having to depend on the DevOps team to spin up the containers or the testers to get them in the pipeline. It sounds like this is almost like a self service solution as well.

We haven't got to the self service mode. Right. You know a bit of things kind of still done on an integration mode like we're doing it but eventually we want to evolve as a self self service mode. Right. That's where the whole concept. This is why I kind of mentioned. I'm actually seeing this as a kind of continuation of that discussion of like testing cloud. A lot of people ask like how do you get that thing? Right. So this is where kind of like this can evolve as like the how we do the clusters in cloud, right? The clusters have everything. It has the framework, it has the infrastructure, is the reporting. It's just a testing cluster. You just upload your test script there and the cluster can take care of running it in pipeline mode.

Gotcha. So you know you mentioned four areas when you were talking about qaops. I think we pretty much covered the tools. You talked about data. So how is data different when you're trying to get QAOPS within the DevOps type of pipeline?

I think when we talk about data there are two part of the data. One is the data that goes into injection mode and the one is the data that comes out of the test which is your reporting mode. Right. Let's talk about the testing the injection mode like your test data piece, right? The test data piece is something we have to think about like is right now test data, how to generate data in volume. Because what this pipelines will usually do is like create enormous amount of execution capacity. Right? And you are running 24 into 7, right. So you need a lot of like you know, reusable data to be created or you have to know when the data is kind of used up and you have to kind of refurnish the data. So that capability of test data has to be done. And I have mentioned something my post and stuff like test data automation need to be talked about. Right? Tested automation. Now how you Generate. Not you generate the data only manually. But how do you generate like automated way of generating data and data which is business process oriented. So this is the first piece, the injection piece. We have to think about it. The second piece is the reporting, right? The data that is coming out. Like you have a pipeline and you have an automation. So you have pretty much nailed down the execution part of it. Like you are not doing anything manually. But where are you then spending your time most on. On analyzing your failures, right? Because your time is going on debugging and getting to the root cause. And that's where you need to think about tools like a reporting tool which can bring up data from different sources. You may be running like in a web test or a API test or a mobile or anything else. Like maybe you are running a load test. How do you see everything in one single pane of reporting, right? So that way you can slowly get everything and you need to have history in it so that you can get the whole holistic view and cut down your time of your reporting. And glad we have a kind of also a solution on their space. Like we call it Digi Dashboard. Okay. So that's something that we do on the reporting side.

All right. So test data is very important. I totally agree. So does Digi have a Digi test data solution then or is that handled by something else? And you act like a framework that these different solutions would plug into.

So we haven't got into the test data yet. Like the solutioning part of it. Okay, that's something will be coming up in our upcoming years. Firstly we wanted to build this pipeline and the reporting, right, these two things and then you start kind of creating that need of this volume data. Once we get the volume data need and then we will kind of build a solution around this. But there's a way we kind of like propose our clients and customers to kind of start this process, right? And we propose to start from the reporting because when you start reporting, you can actually start figuring out the efficiency of automation. You have invested in automation, people earn money, right? But you need to know, are they like, is this automation stable? Is it finding quality issues in your system? So that's why we created the reporting dashboard, which is called the DG dashboard. And the most important features of those are kind of it's a universal dashboard. Like you know, it can take data from different frameworks, different infrastructure. Like you are maybe in cloud or a vm, you may be using a cloud providers, you can take data from there. The second is it can do enterprise. It can collate data from multiple sources and kind of show that you can filter it dice and size based on that particular team. Right. And then it's a real time. And finally it's a cloud based so you don't need to think about like you know, how my capacity of storage is going to be. Like, you know, you can do it optimally. So that's the first point. You start getting those stats and the stuff then you kind of moving to the doing something is not working, right. And you figure out and build that pipeline so that you can build the efficiency from there. So that's the second piece. And then we will go to the test data.

Nice. So with the Digi dashboard then you know, these KPIs people never know really what to look at out of the box. Does it let you know, hey, this test we've been running fails X amount of time and it only has coverage with you know, 10% of your application. Is that that type of detail you can get out of it?

Yes, exactly. That's the kind of detail you can get out of it. We allow you to build custom like metrics like because these are enterprise level things. It can be used in small organization, even in larger organization. And every organization has their own need of metrics, right. One organization like Dick came and said, we use a lot of vendors, right. And we want to know like how are these vendors performing in those certain areas, right? And then we had the metrics which is specific for that. Right. And some organizations said, you know, I also want to see a metrics of a time, right? Okay. That what is the time taken over the time for a customer journey, right? A metrics of that. Right. So we have that metrics as well, right. We also have that kind of the granularity of something fails all the log, screenshot videos and everything together. So this is a more Persona race. Like you can get the metrics, which is stats as well as you can go and see like, you know, a tester would need all the details to get from a issue to a root cause and fix it.

So someone's trying to envision this. How is this different than say a browser stack or a lambda test which is like a test environment in the cloud. How is this different?

I mean very good question. Like we got to understand the tool segregations, right? You know the thing. So when you look at browser stack or let's say lambda test, right? So they are going to report only on their system, they only going to show what has happened on there. So what we do is we over the time we have more than 10 partners and they're like, who are building these solutions together. Like, so what happens is, I'm just going like I had a discussion with Tricentis, like Tricentis being such a big organization, we are also kind of partnering with them. So what happens is no matter whether you are using BrowserStack, LambdaTest, P Cloudy or Perfecto, like any one of them, we gather the data and still show the data. For an end user, they don't need to worry about like whether they did. They just can see everything, the logs in there at the same time. Like whether you use Tricentis or you use Cypress or you use Selenium. Right. You can still see everything in one place.

Okay. All right. Definitely. So I'm just thinking like when you go in front of your executives or something, they don't necessarily care. Oh, this is what happened in swat. This is what happened in Blood. It's almost like a high level executive go to QA ops dashboard of everything that's going on within the organization almost. Am I understanding that correctly?

Exactly. So we call it, it's a Persona base and three Personas are there, Right. One Persona is a directors and above. Right. So they care about what is the state of union of quality. They don't care about logs and stuff. They care about is the organization. Big programs are working fine. Right? Okay, Is that completed or not? So we give that view, organization view. Then you talk to the managers who are more focused on their side of the story, their teams and their things. Okay. We allow them to cut and dice the data based on that. And then again, finally we have the people who are also developers and testers who are trying to figure out what exactly failed in their test and get to the root cause. Right. So we give all these Personas so everything can be in one place. And you don't need to keep maintaining different spreadsheet or different kind of reporting system. We just bring everything together.

So how extendable is your solution? Because it sounds like you want to work with everything. So do you have an API that people that can interact with or do you have to create accelerators or something to get something integrated with your solution?

So the whole process that we think about is to acceleration, think about the build versus buy thing. I can have a tool and then build the integration and do the stuff. How long does it take you to be operational? And even if you do that, it can only be used in one team. But how can you to the enterprise level? So what we do is like we get the tool, like we help you with the integration. Right? The integration is we take care of it and you get that everything up and running within the shortest span of time.

Very nice. So how long is that process then?

Usually like we take like two to three weeks to kind of get the organization startup. Like imagine you're running on a enterprise reporting dashboard. Like I'm in the industry. That's why we realized that reporting is a big challenge and there is not much focus in that area. Like how reporting can be met faster. And I give you some stats like, you know, if you do reporting faster and get to the root cause like faster, you'll probably be saving 30 to 40% of your time of analysis and also developers time.

No, I definitely agree. I have this conversation for a while where it's called Meantime the diagnosis almost where a test runs, you have a hundred failures and the next build's about to kick off in an hour, you have no way to go through all 100 tests. If you had a way to triage where it says, all right, 99 of these tests all due to the same exact error and this other one's a real issue. It just allows you to triage quicker. So does your reporting do that for you or does it just highlight all the different areas? Does it actually do any type of automated triage for you?

It does allow you to because it captures not only the test, it captures the actually the trend as well. Right. So what happens is you're seeing one test case, but that test case has run maybe five times, 10 times, right? So what happens is once you can see all the 10 results in one place, you can instantly make that call whether it was a flaky test or, or was a stable test. If it's a flaky test, instead of analyzing it like you know, fix it first, right? And then start looking at the error messages on those things. Right? First fix it, bring it back stable. Because that is not giving you an application issue, that is giving you like automation is not stable, right? So that piece is gone. And more and more we are also going into that direction. Like we are first getting that data martial with the organizations that we are trying to work with, we are trying to get them their data in one place. Then we can try to analyze and give them the direction that you know, instead of like doing a search, can we recommend you that you should be looking at this particular failure because this has caused problems in production. Maybe this is more function like more business critical. This may have other issues side Effects as well. So we are going to give you that recommendation as well. Like we are not there. But, you know, that's something we are trying to build. As we build the data, we are building that intelligence that you know how to kind of also analyze the data and get to that.

All right, so speaking of intelligence, a lot of data people must ask you all the time, what's going on with AI machine learning? Is that something that's built into the product? Is it on your roadmap? What's your view on this?

Absolutely. Like, you know, I'm glad you asked this question. You know, I get asked this question every time when we talk about data. Right. So I'll break it up into two part. Okay, so two things. We are trying to leverage data here, like the AI capabilities here, right? One is analysis, right. Where we should focus. Okay. Like, because there are so many failures happening, some things, we want to give you that recommendation where you should focus. Now, once you get to focus, what you need to look at, we are slowly trying to get to that point that we can get you to the closest root cause of that, right? So this is where we want to kind of get that information and AI, right, because you need AI here, true AI, because it cannot be done by programming. Because every day and day out, like there are so many error things which are happening, new errors are coming into the thing, right? And you also have to understand, like, you know, once you do a analysis, you have to have a feedback loop and saying, like, what did this analysis? And this is the root cause. So models have to pick up that information. So we were trying to get to that point that we call it DG predict, right? Okay. It should be able to kind of do the predict, like what may be the root cause had happened, right? So that's where we want to kind of go towards that. Coming to that first, recommending you what you should be looking at and also trying to get you to the point, what is the cause of it?

Before we go, is there one piece of actual advice you can give to someone to help them with their DevOps QA ops efforts? And then what's the best way to find, contact you or learn more about your solution at Digi4.

So I think, like, what we, I would mention here is like, yes, we talked about technology, right? But to bring technology, and bring technology in an enterprise like, you need always guidance, mentorship and collaboration, right? So everybody in the industry who is trying to kind of adopt new technology, where there is a challenge or there is kind of resistance, reach out to the industry. Okay. Reach out to people around you. Right? Okay. And have a conversation, collaborate and see how people somebody has done it. Maybe you can find a solution there. Like you can have a conversation which can help you in solution. So instead of having silos and trying to kind of do that solutioning yourself, reach out to the community and we have an excellent community. I'm thankful for Joe for kind of like keeping us always so connected. So this community will help you to kind of adapt and that's where together we can bring a positive change in the industry. And to your question about like how they can connect. I'm very active on the social media, especially on the LinkedIn. You can always connect me. Like you can type my name and it will come up. I do have another like note on the thing. One floor CEO. I'm the CEO of Digi4 but you know that's another tagline I have. Reach out to me through LinkedIn. That's one thing. And always you can connect through like as well as DG4 connect to us in DG4 we can have a bigger discussion about like how we are using cloud based QA ops to change the ecosystem of quality engineering.

And for links of everything of value we covered in this DevOps toolchain show, head on over to testguild.com p11e and while you're there, make sure to click on the SmartBear link and learn all about SmartBear's awesome solutions to give you the visibility you need to deliver great software. That's smartbear.com so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test skill.info and let's make it happen.