From Chaos to Clarity: Improving Software Testing Practices with Prathyusha Nama

In a land of testers far and widely journey seeking answers, seeking skills, seeking a better way. O the Test Illumination Testing Podcast Guiding testers with our nation awesomeness. From ancient realms to modern days, they lead the way. Oh, the Testicle Automation Testing Podcast.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Pradusha all about From Chaos to Clarity Navigating Software Mishaps with Effective Automation. We're basically going to cover the Microsoft CrowdStrike incident that happened a few months ago. If you don't know, Pradusha is a distinguished leader in the field of automation with over 10 years of hands on experience and a deep specialization in mobile automation. Currently serving in the Quality center of Excellence for Test Architecture, she plays an instrumental role in advancing testing practices at our current company. So stay tuned to hear a really important topic that I think is going to change the industry as it applies to testing. You don't want to miss it. Check it out. Hey. In today's app driven world, businesses must ensure flawless user experience. But testing can be a challenge with limited device coverage, slow and reliable setups, and fewer automation options. And that's where Today's episode sponsor BrowserStack comes in. Their App Automate solution solves many of the challenges with its cloud based app automation platform. It's simple to use and packed with features for confident app releases. With App Automate, you can integrate test suites in minutes via the browser stack SDK, run thousands of parallel tests across 20,000 plus robots. Real devices easily integrate with CI, CD pipelines, test frameworks and more test apps and internal and stage environments even behind firewalls and debug instantly with video recordings, logs and AI powered error categorization. Their updated Build Dashboard also features advanced debugging like flaky test detection and quality profiles. Plus with their Real device cloud you can test key use cases like camera injection, QR barcode scanning and simulate time zones. GPS and network conditions. Boost your app quality and competitive edge with App Automate, your one stop solution for scaling mobile app releases. Give it a try. Support the show using the link down below. Hey, welcome to the Guild.

Hey Joe, this is Pratyusha. It's really an honor to meet you. I've been following you on podcast and everything, but it's really nice to see you in person and be on the show. So thank you.

I appreciate it. Thank you. Thank you. So I guess before we get into it, really what caught your attention with CrowdStrike? Because I know you Contacted me, said, hey, I like to cover this. What was it about it that made you go, maybe this is something we need to dive into a little bit more?

So especially for like whatever. If you see lots of industries nowadays, like airlines or all other companies, they're outsourcing most of the testing practices to a different vendor, a different company. So it's like there are a lot of integration gaps and if you see that lot of emphasis is not towards the testing. So most of the times we see that the focus on the UX UI or the development part is not the same with the testing. So these types of incidents where, you know, entire world is in chaos, like something is broken down, you are just getting an error message, like airlines are unable to do what they're supposed to be doing, everybody is in chaos and just not understanding what's going on. But then if you dive deep into what could have caused the problem, it kind of, you know, for people who are already in this field for quite some time, you can kind of make wild guess on what could have gone wrong on both parties. Like it's not anybody's fault, to be very honest with you. And also the fact that it's very appreciative of both CrowdStrike and Microsoft, especially with the fact that they've kind of taken a quick route, the turnaround time, the way they were able to fix it, making sure they were in contact with the customer base, making sure they're providing updates, that is all like really good. But now it is. I think this incident has made a lot of experts, companies, everybody, to kind of reevaluate their whole testing processes. So that is something that I always was interested in, especially throughout my career. I've seen that happening quite a bit where the funding is not that good enough for testing side of the projects usually. And most of the times we also have a notion that one person can just kind of, you know, do the whole job like it. I mean, why do we need a person to just see if it's working or not? It's not that simple is what I really wanted everybody to be aware of and focus on the importance of testing. Because even in our day to day life, the very basic thing, you wouldn't buy something without you visually see it, touch it or some way test it, right? So it is the testing is part of our daily lives. So I really wanted to emphasize how important testing is for any industry, not just software, for all of the anything that you're putting out there, it is mandatory that we just have to ensure it is all like working up and running and all that. So this, this topic blends with all the things that we would otherwise talk on different occasions. But all of it comes into picture when we look at incidents like this. So that kind of caught my attention. I really wanted to like talk about that.

Nice. So could this have been seen? Like how like just. Or does it highlight like all the interdependencies we have on software, not knowing that our software relies on this other piece of software that if that goes down, like, who would have known that airlines would have been affected? So was this avoidable or do you just see as something we now know about? So now we need a plan based on the information we now have.

It could have been avoided because we have lots of testing that we do, right? Like if you follow the best practices like contract testing, you have shift left, you have backwards compatibility testing. So it's basically the fact that the emphasis on the integration testing as well as how soon there should be a process of, you know, the company should follow a certain process. Like it's not that simple where you would just update something. So the issue is basically, you know, the contract between what the software system on Microsoft, which is the client is expecting, versus what the CrowdStrike security update was providing. So it's basically that the input parameters that were provided is not something that Microsoft was expecting. So this could have been avoided if they had contract tests in place to make sure when they do this initial deployment on their like dev or staging environments that this is this working fine if there was a backward compatibility testing in place, any new updates, making sure they don't break the existing version of all their client systems. And of course like Shift left practice that is very popular these days where all the companies are kind of moving towards shift left to be able to identify issues sooner rather than at the very end of the game where it is like something in, close to production or in production in some instances. So there are a lot of practices that are, you know, that are being done. It's not something new or this is not something that just came out of an incident. But these are the things that have already been there, companies following it, it is all being done. It's just that the importance of why it should be like implemented as part of our process. How strong can we make this testing process is what it all gets down to. So it's just that it could have been avoided. Especially with when there are a lot of vendors and a lot of outsourcing that happens. It is really important to make sure we have like strong practices in place. And all of it, like none of it is actually to be done manually. Like you wouldn't even expect that something should be verified manually. You have a lot of automation improvements, you have AI, ML, everything. Like we are trending towards a different technical, you know, a lot is happening so it could have been avoided. It's just that the emphasis on the implementing strict policies for testing strict in place, like the release plan for anything, even if it's a minor update, especially for cybersecurity firms, it is really, really important that they focus more on the testing practices and procedures.

Absolutely. So before we go any further, just in case the crowd that listens to this is pretty tech savvy. But for those that don't know what is contract testing, because you mentioned it a few times as something that probably could have helped in this incident.

So to break it down like in terms of, you know, let's take an example of API. So you have API provider and consumer that's going to use these APIs, right? But they are. The development of API happens on a different team and the team that consumes these APIs is going to be different. So how do you ensure, like at the beginning of the project you would say okay, as per the requirements, my API is going to just give you this response. It's going to have X, Y and Z as mandatory fields and something else as optional. That's the requirement use case for producer API. Now on the consumer side, this is all communication. Like at the beginning of the project they would agree, okay, this is what it is. And then once the consumer team starts developing their application based on what was discussed on the original implementation, details like X, Y and Z as mandatory and something else as optional, they will develop off of their own teams. Now imagine there is no contract testing. Something has changed on the production side of the API. They have decided now to add one more mandatory field but they forgot to notify or they don't have communication happening between the consumer and the API that is being developed. So then the consumer team just ends up releasing their product and then once the API deployment happens, it's not going to work in production because not all the times you have staging or some environment available before you go to production. Sometimes API team does their own release schedules versus the client applications stick to their own schedules. It's just that if there is no contract testing, it would be way too late by the time they would identify that something was broken because they did not stick to the contract that was agreed upon. So these contractors are usually run at the beginning of the developmental cycle where both consumer and producer API would have contract tests written on their end where they would make sure the expected response is what they're getting from consumer side and vice versa for the API that is being developed. So these types of tests we have lot of, you know, the PACT bi directional flow with pact. So there are some popular contract testing tools that are being used by majority of companies. So this can also be leveraged for not just APIs but all other types of testing as well.

Nice. So yeah, it sounds like it's a great thing to use if you have a lot of third party dependencies on your application. We don't actually need their application, but you can, you'll know what kind of responses that they and calls they're making.

Yeah, nice.

All right, I want to dive into a few other things, but just in case I may have missed it, I forgot. Do you know what caused the CrowdStrike outage? What? Eventually it was. Was it a testing issue?

I think yeah, it was. So it's basically like there was an update on CrowdStrike software version. Like they had these parameters added. There is one new parameter, but then looks like that was not. They did not take into account what could have been the possible effects of this update. So they just went ahead and updated it. Microsoft on the other hand was not expecting that update. So this version of Microsoft was not compatible with the version that was deployed. And that's kind of, you know, causing memory out of bound exception that's eventually like crashing. So you just get an error message on the screen when you try to access the applications. So that was like. Yeah, it is a testing issue. Yes, gotcha.

So Microsoft, it's an os. How would they have known? They must have so many third party dependencies. I guess it goes back to what you said. You need to have a good testing procedure or testing process in place. So let's start there. What do you think would be a good testing process to have in place? Because your title is Quality center of Excellence for Test Architecture, which tells me already, already your company takes this seriously. But for some reason I'm seeing Quality Centers and Centers of Excellence disappear. So what do you think companies need to have in place in order to have a strong testing process?

So there should be, you know, like a process for any. Like for example, on Microsoft side there is no way for them to know that the CrowdStrike was going to deploy a software version and that would break it. If they had known, they wouldn't let this happen. Right. So of course the issue is also with the fact that they, the cyber security like the CrowdStrike was given access to the kernel of the Microsoft, which is like, which is where you have lot of your, you know, important stuff. There is no way for if something goes wrong, it's going to crash. There is a reason why we do that because all the cybersecurity stuff would need that level of access to make sure we are handling the hacking and all that stuff. So especially for these types of things, it is more important to have strict processes in place. Like I was talking about the compatibility testing, making sure they have some types of testing that automation tests even that would mean very critical ones that they run every like, you know, probably every day to make sure if something changes, even if it's very minor update on the dependencies that they have, as long as they can identify it sooner. And then, you know, if they had some tests in place before the upgrade would have happened, they would have seen an issue with minor upgrades or something like in terms of what if I have a version 1.0, but then I'm getting a version 1.2, if it was failing, they would have caught it sooner and they would have, you know, they would have had that collaboration with the CrowdStrike beforehand to make sure. This is what we are noticing on our side. I think we just have to make sure that is kind of being taken into account on the test that you have written on your end to make sure it doesn't happen in production. So this here is like, you know, the lack of coordination in terms of testing. Right. Like there are certain tests that CrowdStrike runs. It's not that they are just, you know, doing the migration. They do have tests that they're running. It's just that they did not take into account like what could have been the effect of, you know, if a system or one of their clients has a version that is two levels below the version that they are putting out there, will it still work or would it cause any problems? So that is like making sure your backwards compatibility is also in place. So it's like it's not one place where this process needs to change. It's basically that everybody, like the consumer, CrowdStrike, Microsoft, everyone needs to have these processes in place. Because most of the times you would think that, you know, even for our regular company release cycles, for any feature or anything that goes out the door, we have multiple levels of multiple rounds of testing that's done before it gets to production, like probably Months of time because you have, you know, your Sprint level testing, you have regression, you have integration, user testing, all that and then you do production smoke test before you make it live. And also the fact that especially for these types of huge upgrades, we could also have, you know, rolling out like specific set of users. Instead of rolling out to the whole user base, they could do a, you know, selected rollout. Like for example in like for companies that deal with, you know, doctors and patients, like my company, for instance, we have different regions. It's not going to be in like the same country. We have to be HIPAA compliant. We have FDA regulations that are different for each country. So what happens here is that instead of rolling out the feature for the whole doctor or the customer user base, we do selected rollout. Like hundred people. How is it going? Is it fine? Do we encounter any issues? Because at that point instead of affecting millions of users, you would just have hundred users that are affected by worst case if something goes wrong. So that type of strategy is something that you would, you know, you need to have in place. Also the firsthand testing that happens immediately after your software is updated, right? Just to make sure you have rollback plan in place. Like something goes wrong, you immediately can find out something's off. You immediate, immediately roll back to the previous version to make sure if it happens off hours, like nobody is going to like at least the intensity of the issue or the affected user base would be very less. So these types of processes, it's not something new, it's just that it's all already there. We just have to have that being documented or the gatekeepers. Like for example, we talked about center of Excellence. The reason why center of excellence is formed is to make sure we are following all the best practices that are good in terms of software testing. Making sure we follow everything before products get out the door, making sure it's bug free. What else can we do? How can we improve with whatever latest trends that are coming up in the industry? Making sure how do we incorporate them into our release cycles? All of that because it's equally important for our features to work. Even if it's a cool feature, if it doesn't work once it gets to the production, there is no point.

So yeah, I love that answer because now you're talking about shift, right? And like feature flagging, canary testing that probably definitely would have only like you said, roll it out to a few users or a hundred or a segment and then be able to roll it back would have probably solved this quickly. So that's a great technique.

Yeah.

How hard is it to get started with that then? Is that must be like a big commitment, right? Or are people afraid to do that in production? Is that why you see, maybe there may have been a gap there.

Yeah, possible. Or maybe they never even thought about, you know, they wouldn't have guessed that something would have. I mean Microsoft has been in the industry for like so long. Did we ever see this happen? We did not. So no one would anticipate that. It's just that, you know, now that it has happened, everybody is talking about it. But what if it didn't happen? We wouldn't talk about it. We would just go off our business the same usual way. So that is the only thing that I think could have been the issue because it's just that, you know they have certain checks that they do like the basic checks that they would do on each end. But then the possibility of something like this could go wrong they probably did not encounter or take into account. So now that this happened they kind like everybody would start to think of even the statement that's issued by Microsoft after the summit that happened on understanding what this issue is about, what could have, how could have this been avoided and all of that. If you see most of the things that we've talked about are already being, you know, taken into account. It's just that I feel like it's just that, you know, no one anticipated this would happen. It would be like, you know, as long as stuff is working, all of our critical path to production is all good, we are fine. But these types of things, you know, especially comes into picture when we are dealing with third party applications who we don't usually talk to on day to day basis.

Right. So how do you then balance that? Because you could test for everything. I know a lot of people are risk based but how, how do you balance the need to test versus the need to get in the hands of the customer and be do it cost effectively?

So yeah, like I said like you could use automation, you could use, you could leverage automation for cost cutting. You know, instead of having manual people. Like if you end up needing like four or five people on both the sites that could be, you know, reduced cost to quite a bit by relying on automation. Cicd where you have regular runs, you don't even have to, you know, talk to the, or have a touch base or something with those teams. As long as you both the parties have their critical test running up and running on their CI every now and then they get the results faster making sure. The integration points are all touched, not just you know, sticking to one system. Like for example if you have a two or three teams that are working together on an application, like a UI team, backend team, API team, all of them will have their own tests that are really good that like that they do what they're supposed to be doing. If there is no integration testing happening between these, like every UI test is only sticking to UI checks, API to API and then database to database. Now imagine if they don't do any, any other testing, making sure backwards compatible or something like integration. The end to end flow is all good. Nothing is going to go wrong. We definitely don't see that product working in production unless they have someone who's making sure all the teams are connected and all of the testing happens and everything is working fine. I think that is what is required here as well in order to do it cost effectively. We have with AI and ML there is a lot that is coming up, you know, self healing, automation. We could see and leverage how which parts of it can be used for security because not everything can be used for security because it has a lot of other complications. It's not as simple as API and UI testing but definitely worth exploring and you know, innovation should happen there where we could leverage some concepts of IT to make sure the issues are identified automatically instead of manually someone finding it or something. I mean as long as you know the problem somehow automatically coming to you, fixing that and making sure everything works fine is doable.

Nice. So have you had time to mess around with any AI technology and how has it helped you?

I did it out of my personal interest but nothing that I was able to do it within the current organization because we are HIPAA compliant and FDA regulated company. So it's not you know, even for like when I was on the mobile team we were looking at, you know, trying to explore all the cloud based solutions for running our tests on simulators, emulators especially because we have photo capturing which is AI AR related machine learning. We heavily rely on machine learning for our image processing because that requires testing to be very efficient. With the actual phones and actual camera though we have automation that would actually do this in order to be able to test this out across multiple devices. We tried exploring lot of cloud based solutions and we were like the approval not was not in place because of you know, the security policies and HIPAA complaints. So I was not really able to, you know, apply them in my job. But I out of my interest. I did look at lot of stuff there is one conference that happens every year in North Carolina. SQA conference. So there you have lot of vendors coming in like the browser stack, all of the aptly tools everybody comes in. So there we have a lot of sessions for AIML as well. So how like what they could do. So yeah, I mean whatever the research I've been doing, all the papers that I've published, it's all out of my personal interest.

Nice. So you know a lot of times I think companies see testing as a cost center, not a value add. Do you think this will help? I think I saw CrowdStrike was being sued for billions of dollars. Do you think this helps testing in the long run or do you think this be like any other incident would be forgotten in a not a mother.

So no, I don't think this is going to be forgotten that easily. Especially the fact that you know there is an emotion piece involved in this incident. Right. I mean if it was just computers shutting down, I mean who cares? The people are going to forget. But it's not that. It's basically, you know there were traveling people that were affected. Imagine the emotion that goes on there, on there. And there could have been some people who are probably planning to get married. Maybe they were going to meet their loved ones, their families, you know, they're something that's really important in terms of their life events. Would they ever forget what happened? They missed it because of the airlines being unable to, you know, get the stuff on time. They all missed what they wanted to do. So it's an emotional attachment at this point. Right. So it's not that easily forgotten. And also for the companies it's the fact that the, I mean though the customer base that was affected is very less for the Microsoft user base but it still got nationwide attention. Like everybody were like what is going on? What is going on? So it's not that small of an issue where they would just you know, release one press release and they're fine about it. No, I don't think Microsoft is taking it lightly either. Like if you see the, the things that they are planning on doing the analysis reports. I really appreciate the way both the companies addressed this one because it was never blaming whose fault was it, who was it, who should own it. It was never the case. It was more of like collaboration trying to fix the problem for their customers which shows, you know they really give importance to incidents like this. So I think this is going to set a new tone for a lot of companies where the cost center like you've mentioned most of the times you don't usually get a lot of funding for quality side of the resourcing or staffing or anything. Right. So things would change for a lot of companies. Like no one would want to be in such spot. You know, I think this probably is going to set a new, I think new stage for quality. We will probably have more importance in the future.

Awesome. Okay, before we go, is there one piece of actual advice you can give to someone to help them with their automation testing efforts and what's the best way to find or contact you?

Oh, for me. So actually I also am a mentor on adplist which is a free platform for all of the tech professionals. So if anyone's interested in, you know, finding out more about automation needing help with like getting started on automation leveling up, I am a mentor on adplist. You can always reach out and book a session. It's free, you don't have to pay anything. I have a lot of mentees that I'm mentoring currently who are like, you know, some of them just graduated looking for automation roles. Some of them are like, you know, at the beginning of their automation journey wanting to understand more on how they can get involved and get from like get into senior management roles and things like that. So yeah, that's one way you can reach out to me. You can also reach out to me on LinkedIn or I need a follow.

Up question on that then for your coaching calls then. Is there a common theme you hear all the time that people are struggling with, with automation?

Yeah. So most of the times the things that I fear from them is, you know, the initial struggle of setting it and setting it up. So they have lot of environments, they have lot of test data, a lot of dependencies on other teams. Right. That the struggle that they initially face to just have that whole flow like a full stack. We always talk about, you know, full stack development, but we never really talk about full stack role in a software development testing industry. There is this thing that's full stack. I mean it's not known technical or an official term. But if you talk about, you know, my. So for when I started my career, I was the very first SQA engineer for that the company had for mobile team. They did not have any testing presence in the company for mobile applications. So it was on me to make sure how do I ensure the products software testing. Right. So I was the one who did the POCs to pick up a tool language that would work, finishing the setting up of the frameworks, making sure it's Scalable working across all the environments and then CICD setup and then making sure do I do the DevOps part. So I kind of did the full stack role in my field where I'm doing everything from starting A to B until everything is done and up and running right. But we don't really talk about these types of things. But that is something that is being asked from the people. So that is the struggle that I see from most of my mentees where they are like my job is to just write code but I'm expected to know, you know, cicd, scripting, Pythons, too many languages developer can just stick to one one specific technology and language but for me they ask like Python, Selenium, JavaScript, all other languages, like am I really on the right track? Like do I want to pursue my career in this? Like what do I do? And most of the times lack of recognition is the biggest thing that I see from my mentees that they feel very, you know, demotivated. They don't feel like they're bringing value to the company because any meeting that they go to they always see that the developers are getting credited for product owners are being appreciate because the product is, you know, delivered ux, ui, everything is good. But then for all of that to happen, testers were part of the project but they never really get that credit. So these are like the common things that I hear from all of my mentees.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode head on over to testguild.com a518 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. Joe, my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing the secrets to behold.