Root Cause Analysis Driven Process Improvement

Hello everyone, My name is Linda Westphal from the Software Excellence Academy Powered by the Westfall team. And Mary Jo has asked me to present today my talk on root cause analysis, driven improvement. So root cause analysis, I always like to start all of my talks with some definitions so that we're all on the same page. Root cause analysis is an investigative process used to assess a set of product defects or process problems in order to identify the underlying factors, the systematic causes of those defects or problems so that you're not just fixing the symptoms, you're actually fixing the root cause and therefore hopefully keeping those problems from reoccurring over time. What are the benefits of root cause analysis? Again, you're actually identifying and therefore able to address the underlying factors, the systematic causes of the defects, the process problems, rather than just putting bandages on the symptoms. I make a distinction between correction and corrective action. Correction fixes today's problem. Corrective action fixes the long term root cause so that you don't have those future reoccurrences of similar problems. And that's what we're trying to do, is prevent those reoccurrences of similar defects in our software, similar problems in our processes. It's a root cause analysis is a logical and systematic approach that's based on facts. So you're focusing your thinking in on the facts, the information, and making better decisions rather than just using your gut feel. It also concentrates and focuses on the products and the processes, the services, the projects. It helps eliminate the blame game. Yes, we're looking in the past, but we're looking in the past to identify the root causes so we can prevent reoccurrences in the future. We're not looking in the past to assign blame to the individual. We keep ourselves product process focused throughout and it can promote the organizational learning. Finding a problem one place can. Then if we address that root cause in our organizational level, processes in our product standards, etc. We learn how to solve those problems. We create those patterns that keep us from repeating our mistakes and that creates organizational learning so one person or one project doesn't learn it, the entire organization learns it. And we can apply that to our risks as well. So we're looking at yesterday's problems to identify things that might be repeated as potential risks or potential problems in the future, and that helps us prevent future reoccurrences of those defects and problems. So the difference between correction and corrective action, the way I see it, we have a process, that process is implemented to create Our products, our services, our results. Then we do our various V and V activities on those products and discover problems. Or we have problems that get reported from operations and we correct those problems in our products, in our services. That's correction, also called remedial action. You fix what you found and that rework has an immediate consequence to the quality of that product. Unlike hardware, one of the big advantages of software is once you remove the defect, it is gone, folks. Unless we have poor configuration management and accidentally rebuild back into the product those defective components, which with good configuration management we should never be doing, that product is, that defect is gone from the product, failure will not reoccur. So that's correction. We can do the same thing with our processes, but we take that collection of product defects and analyze their root cause. That's what we're going to be talking about today. And we feed that into improvements in our products, improvements in our processes, and that's the long term corrective action. We keep those problems from reoccurring by fixing our processes or mistake proofing our processes. We feed that into similar products or similar occurrences. For example, if I hold a peer review of my code and the peers that are reviewing the product discover a defect in my software, they're less likely to go make the same mistake. They've learned a lesson, even if it's just that one. And we're going to talk more extensively about doing, rather than just learning individually doing that root cause analysis. So we're learning as an organization now. What are we looking for when we're doing root cause analysis? If we're looking at product defects, we're looking at when that defect was introduced and we're identifying what processes allowed that defect to be introduced. What can we do to prevent future reoccurrences of that defect? For example, when I first started doing user manuals, training materials, I don't know why, but I fumble finger the word perform and make it preform. I switch the R and the E and preform is a word so the spell checker doesn't catch it and it gets all the way out in the field. And one of my customers is saying, what do you mean by preforming the process? What do you do ahead of time to form it? And I said, no, no, no, that's just a mistake, it should be performing the process. Well, I learned that that was one of my common mistakes. Now every time I create a user manual, every time I'm writing training materials, I hit the word perform and I hesitate and I spell it Right. I very, very rarely I have that problem because I've learned to prevent it. I've learned it's one of my mistakes. I think about it, I keep it from happening. I catch my own mistake. So we're trying to prevent those. Well, one way to look at when it's introduced is to use a Prado chart. It's one of the tools for root cause analysis. And what you're doing here is identifying the most prevalent root causes. What are the problems that are the types of defects that are being interjected into your processes. Now, this may be steps in your process. For example, from this chart, we look at requirements defects versus design defects versus coding defects versus testing defects or configuration management defects. We're looking at where defects are being interjected into our processes, and then we're attacking the most prevalent ones of those. For example, according to studies quoted by Carl Uyghurs in his book on peer reviews, 40 to 60% of defects can be traced back to poor requirements to requirements type defects. Well, that's the first level. Then we can take different types of requirements defects and drill on down from there, et cetera. So we're attacking the areas where we're having the most problems and again, putting things in place that help prevent those. Another way of looking at this is a more sophisticated metric of defect containment effectiveness. What you're doing here is you're using your metrics and evaluating the effectiveness of your defect detection techniques in finding and fixing defects in the same phase, the same activity in which they were introduced. So you're looking at the number of defects found that were introduced in that phase, that activity, that iteration, versus the total number of defects introduced in that phase. For example, let's say in the yellow, that was the phase that we detected the defect versus once we've done the root cause analysis, we know where it was introduced. For example, 15 requirements defects were introduced and found during the requirements phase. Five more of them again introduced in requirements but not found until design. One of them wasn't found until coding, three of them wasn't found until testing, and so on. And what we can do is we can look at phase containment, which is simply the number found and introduced in that phase. That is 15 divided by the total number found so far, 25. And we notice that that's we're finding 60% of our requirements defects during the requirements phase. We're only finding, we're finding 71% of them during design, 77 of them during coding. And again, you can look at the numbers here. Again, that's 29 of the design defects found. Out of the 41 that we found so far, 86 of the coding defects were found during code, out of the 112 found so far. So where do we want to attack things? Well, again, notice that we only have 60% phase containment during requirements, where we're in the 70s during design and code. So we want to be looking at our defect detection techniques in requirements, because that's where we're doing the worst job of it. And that's where we would want to focus our process improvement initiatives, because we've got the biggest bang for the buck, the biggest opportunity to improve there. If we could just bring that up to 71% or 70 something percent, we would have some real improvement. And oh, by the way, the requirements phase is the farthest away from being able to test. And if we fix a requirements defect, we're not propagating it forward into the design, into the code, which makes it much more expensive to fix. Then we're looking at what types of defects. The next thing we do with product defects is what type of defects they are. And we're looking at both prevention and detection here. So we're drilling on down. Let's say we have requirements defects, we want to attack those. Well, what types of requirements defects are we having? Missing requirements, ambiguous requirements that are misinterpreted, incomplete, incorrect, contradictory? Our requirements changing and those changes not being communicated. And therefore we're having more problems. And we can actually go back and take those requirements defects, the 25 of them that we've had so far, and drill them down another level of root cause and use something like a check sheet for root cause analysis and determine the types of defects. For example, here our biggest problem is ambiguous defects, closely followed by incorrect defects. So what can we do? How can we change our processes to have less ambiguous defects? Can we train our people? One of the things that we did is we actually created a list of ambiguous words that we were finding in our requirements. An example here might be a requirement that says you're going to process something within 30 seconds or less. Nice measurable requirement. We can measure 30 seconds or less. But what do we mean by process? Now we interpreted that as from the time we put the command in to run the report until the report was posted out on their network to print, it was always 10 seconds or less, no problem at all. But the customer interpreted that as they had the report in hand and their network had so few printers on it, they were way over under capacity on printers. And it could take up to 10 minutes before that report was on the printer. So again, what does process mean? What does quickly mean? You know, it's not measurable, it's ambiguous. What quickly might mean to a designer may be very different than what cwicly might mean to the customers or users. And in fact, to meet that requirement of less than 30 seconds, it's out of our control, their network and their printer under capacity. So we had to actually attach a dedicated printer to our system to be able to get the report printed within 30 seconds or less. So how can we address those, keep things like that from happening again? We start having templates for our requirements. So if it's a measurable requirement, the template now says when this event happens, it's going to take 30 seconds or less for the software to do this measured from this point until that point. So we've corrected our requirements template for performance type requirements to add into that specifically what we are measuring from where to where, from the time the command is entered to create the report, until the report is posted to the network, or until the report is sitting on the printer printed, ready for them to use whatever that made a less ambiguous requirement. We corrected those kinds of things. We didn't have those problems again. And in fact, we actually moved on from there and created a, started collecting lists of words that are ambiguous, not measurable, not finite, et cetera, potentially poor words and requirements. We created a tool that then took that list and highlighted in the requirements document all those words. Then during our peer review, we could decide because not all alls are bad, but they're always questionable. For example, the computer will respond to all commands, all user commands within three seconds or less. Folks, you know what all your user commands are, you're writing the software. But a term like all, when it refers to accepting all credit cards from all banks, that's again outside the scope of our system, the boundaries of our system. We can't implement that requirement, we can't test that requirement. So we would highlight those words and review them during the peer review and correct the ones that were wrong. We identified a much earlier that way. So that's the next step. When was it detected? And what we want to do is we want to move that left, I. E. We want to find problems much earlier in the life cycle. Now, this goes back to the containment concept again that we talked about that metric. But you don't have to get that sophisticated. You can just look at the number of escapes from any given process. So if we're doing software development again and Focused on the product, we can. I almost like to think of our defect detection techniques as screen doors that are keeping bugs from getting into our house, our products, right? So during requirements, we find a certain number of yellow requirements bugs. They get caught by the defect detection techniques. Those are not escapes. But when we do our design defect detection, not only do we find some green design bugs, but we find some yellow requirements bugs. Now we've got those escapes. And again, if we want to go further, we can look at defect detection, phase containment, and you know, we've so far had, what is that, 11 bugs and we only found eight of them. Why did we not find the other three? What can we do to our requirements defect detection techniques to improve the probability that we find more of those or that we find the three that escaped. Everything we find in coding, escape from design or the green design bugs or the yellow requirements bugs. Of course, anything we find in testing is an escape. Anything we find in the field is an escape. So we can just start looking at our defects that escaped. Why did they escape? How can we improve our defect detection techniques so that we don't have those defects in the future? So not only can we look at phase containment, but another metric we can look here at. Here is all of the escapes, not just the ones that were introduced in the phase. And this is defect removal effectiveness. So you're looking at all the defects that were found and removed during that activity, the defect detection activity, divided by the number of defects present when that activity occurred. Now, just to show you a similar chart now on the left, instead of the major activities like requirements, design code testing, now we're looking at each defect detection technique and how many defects did it find versus how many escaped. And you're looking at all the defects that were present at that point in time. For example, requirements, you can only find 25, you found 15. Phase containment, defect removal, efficiency are going to be the same for your first activity, but as soon as you go to the next activity, you've got the ones that escaped from the first activity. So not only can we find the 41 requirement, I'm sorry, design defects, the 41 design defects that were introduced during the design process. But we can find the 10 requirements defects that escape from the requirements process. So we could have found 51 the 10 requirements escapes and the 41 design defects, but we only found 5 of the requirements and 29 of the design. So we're looking at a different percentage here of all the escapes. Then when we can look at what components are defect prone again, At a minimum, these are more error prone components. They have a high probability of having more defects in the future. There's lots of empirical evidence folks, that defects cluster in our software. I like to call it the cockroach effect. You don't see any cockroaches in your house. You're happy, no bugs here. You run all your preplanned test cases. No bugs here. We're happy in this feature. In this component, you see one, you kill it. It was an aberration. It came in with the groceries, it snuck into the house somehow. But by the time you've seen your third or fourth or fifth cockroach, you know they're in the woodwork in your house and it's time for some real debugging. The same is true in your software. That's that Prado principle we talked about already. Again, 20% of your code is gonna have 80% of your bugs. So you wanna identify those error prone components. They're candidates for potentially completely re engineering them. For whatever reason, they're overly complex, they're old legacy code that's spaghetti code and wasn't built under good structural design to begin with. For whatever reason, they're error prone. We may want to re engineer them. At a minimum, they're high risk components. So we want to test them more. We want to dedicate more of our testing resources to the error prone components than we do to other components. Of course that means for me that the test cases for that are going to be some of the first candidates that go into our automated regression testing. Because if it's been error prone in the past, it's gonna be error prone in the future. Of course I put other things in there, like the most used software or the most mission critical software. Anything that's safety, security related. Anything that is mission critical, for example, as simple as a word processor. I submit to you that the save command is mission critical. If all else fails, you've at least gotta save be able to save what you've done so far. So again, error prone components, they are candidates for re engineering. For example, go back to the Pareto analysis. But now we're looking at defects in the components rather than in the process. You know, where they were introduced. We're just looking at where they live. And module A has the most, module B has the next most notice. All the others are way down here in the minutia area. So we're probably going to abc, maybe even D put those into our automated test cases, identify those as error prone components that are higher risk that get more testing. Now we can do this even as early as, for example, our peer reviews. We looked as we were automating peer reviews, we looked at deeper vex per thousand lines of code for those components, those modules, and anything that had over a certain amount was definitely a candidate for reinspection. Again, higher risk after you've corrected things. We want to peer review it again because inter again, if it's been error prone in the past, it could be error prone in the future. Simply correcting those defects may introduce additional defects. If it's really high risk, we would investigate it for potential re engineering early in the life cycle. If we build a component that's got problems, we may decide to re engineer it now. The other thing we can do is look at similar components, because if that component has those defects and there are similar components, they may have similar defects. So we want to investigate other components. For example, if I find that adding a user to the database has a lot of defects in it, I may want to look at deleting a user from the database, modifying a user in the database, searching for a user in the database. Those may be similar types of components. Or if I find a module that has a certain type of looping or a certain type of sorting structure, I would look at other components that had that same looping, nesting, looping or nesting that same kind of sorting structures. If I've made the mistake once, I can make it again. And that may help me identify again defects much earlier in the life cycle than not finding them until the later testing cycles. So that's product root cause analysis. We can also look at process root cause analysis. Of course, we're looking at the processes and how the process might have an issue as we're building that product again, we do correction. We fix that step of the process, something's broken. We do immediate remedial action to fix that. But we don't stop there. We take those process issues, learn, look and do a root cause analysis of them and feed that into improvements. And again, it may be improvements in the process or even in the product. Sometimes process issues can help us solve product problems, make the process more error resistant. We can buy a compiler that we forget during the process to initialize our variables. We get a compiler that initializes our variables for us and fixes our products. Of course, we can also come up with process issues because of audits. We discover non conformances and non compliances. During those audits, we do root cause on those. Or there may be other process defect detection techniques or problem Detection techniques. This may simply be a suggestion mechanism within your organization. It may be some kind of tiger team or some kind of process improvement team quality circle that is investigating our processes and and identifying where there are process issues that need to be corrected again that feed into correcting our processes in particular, but potentially even some of our products. Now when we're doing process improvement, somebody's identified the process issue and we've opened a corrective action request in the product. It's typically a problem report or a list of defects found during a peer review, reports of problems from the field. Here it's some kind of issue, some kind of suggestion, some kind of identification of a process issue that needs to be corrected. We open a problem report typically called a corrective action request or a capa. A corrective action preventive action request. The first thing we're going to do is determine the root cause of that. We don't want to just band aid the process. We want to develop its root cause from that. The quality action team that's in charge of fixing that process would develop a corrective action plan for that. Corrective preventive action plan for that. Correction is going to correct the problem. Prevention is going to keep it from happening in the whole organization, not just where it happened. Correction fixes the today's problem and the long term solution to that in the current project, the current process, the current product versus long term, keeping it from happening again. And you want to do both. Typically if it's an organizational level process, and that's where you get the organizational learning is to have those organizational level processes we are, it's going to go to some kind of approval authority. You just don't want people changing corporate processes without getting approval again. The corrective action team could come up with a plan. The approval authority says that's a great plan for your project, but that's not going to work at the corporate level. Let's use you keep it as tailoring for your project or they may approve it, turn it into organizational learning and implement an improvement across the whole organization by improving the standard organizational processes, the standard organizational work instructions and in fact folks, the process tells you what to do. The work instructions tell you how to do it. They these are things like guidelines, templates, forms, training materials, whatever that checklists. In fact I find that templates and checklists at the organizational level are my two most improved. The things I improve most often I find a defect. I put it on the checklist so I don't forget to look at it next time. Like the Preform versus perform. I didn't initialize my variables. I put that on the checklist. So we check for it next time. And folks, not only do the peer reviewers use that checklist, but the developer, the author of the work product should be using it as well. The person that's implementing the process should be using that checklist. If they forgot a step, put it on the process checklist so you don't forget that step next time. If you forgot something, for example, you forgot to define your safety requirements, or you forgot to define your throughput requirements, put it in the template, have an actual subsection for that with examples or templates for doing those. I have templates for not only my requirements document, but templates for different kinds of requirements. Now, a simple one for a functional requirement is when this happens, the component or system shall do what. I see a lot of requirements that says the system shall do this, but it doesn't tell you when or under what circumstances or what event kicked it off. And then it becomes ambiguous and may be misinterpreted. Having those templates of reminding people you gotta have the event, the circumstance, the state that causes that requirement to be initiated. So you can have a more complex one that says when this event happens, if these conditions are true, then this system or component shall do this. And I've got those for a lot of different types of requirements. Now I don't lock myself into them, but it's again, you always wordsmith afterwards and make the sentence as good as you can get it. But the template at least gives you a starting point so you don't forget things. So those are the kinds of things many times come out of these improvement plans. And if it gets approved at the organizational level, you put it in, you implement that process improvement at the organizational level, if it's kept at the project level, you implement it at the project level. So whatever that improvement is, that improvement team may be different than the action team that came up with the plan. Again, you may have subject matter experts or process experts that define the plan, but the people actually doing the process are the ones that are going to go implement it, whatever is appropriate. If it's a change of any size or significance, I also like to pilot them first because just because you think it's a good idea doesn't mean it really is in reality. So get one project, one product, one team, whatever, to pilot that improvement and then do additional analysis saying, was that improvement successful? One of the reasons we want to do incremental development and do an improvement at a Time. Because if you do five improvements, two of them may have helped, two of them may have made it worse, one of them may have had no impact at all. But you out, you look at your process measures and it looks like nothing improved anything. So we want to do these one at a time or maybe two, but so you can distinguish between them. Again, if it did improve things, it did get you to the level that you wanted to be. Then you go propagate it, institutionalize it across your entire organization. On the other hand, if it didn't work, you go back and do more planning and feedback through. If you propagate it every place, you again have to do the analysis at the end to make sure it worked every place. Because it may not have. If it doesn't, you may have to go back and do replanning. If it did, you're done with your improvement process. But at some point your auditors or your reviewers, if you're reviewing it yourself instead of having independent people do it, verify that that control system is in place, you have institutionalized it and you're prevent, actually preventing reoccurrence. You're preventing future occurrences of the same issues. So that's your basic improvement on your processes. Now examples. There's many ways of identifying process improvements. You can perform root cause analysis on one or more product defects, on one or more process defects. You can look at dissatisfact dissatisfied customers. You can look at negative trends on your control charts or any of your other metrics. Folks, metrics many times tell you there's an issue long before your gut feel can see it. Non conformances or negative observations out of audits and reviews, identification of poor performance during process assessments, identifications of wastes, if you're using lean processes, et cetera. Now what are some of the ways I gave you the picture above and just had that box that says do root cause on your processes. Well, what are some of the tools or some of the ways of doing that? One of the classics that comes from the 1940s and 50s is your cause and effect diagrams, also called Ishikawa diagrams, also called fishbone diagrams. What you do here is you put the problem in the head of the fish. Ineffective defect detection processes. And then a lot of people call them the four Ms. Or the six M's. Management, manpower, methods and machinery. There's the four M's. The other two are materials and measurements. Although materials and measurements in software rarely cause problems with your software. So I normally don't include those and I change manpower to people because I'm female, what can I say? And I instead of machinery, because we don't have a lot of machinery, it's not like we're doing solder wave machines or whatever. So I call it the environment. Are there issues for the test bed not being good or the compiler being the wrong version or whatever? And then you literally just brainstorm or work with your quality action teams, your quality circles to identify and talk about potential causes. And then you go investigate each of those, maybe by using one of those check sheets that we talked about earlier, to see how many occurrences of each of these are actually happening that are causing ineffective defect detection. Another way of investigating root cause with a cause and effect diagram for processes is put your process issue or problem in the head of the fish and then along the backbone put the steps in your process. For example, if you're having ineffective code inspections or, or requirements reviews, you might put the steps of the process, the planning step, the preparation step, the meeting step, the follow up and corrective action step along the backbone of the fish and then have your root causes branch off of that. Now the previous one, I only had one branch, folks. You can do as many branches as it takes to drill yourself down down to an actionable root cause. Now those are great for us left brained people, our analytical people. I find that people that are more right brained, that are more creatives, need a more creative solution. That this almost becomes too logical and therefore too restrictive for them in many cases. So instead you can use what's called an interrelationship digraph. A digraph is simply a directed graph. So again, we have ineffective code inspections. Well, why? Because there's too many defects in the code. Well why? Because the coders don't desk check it or the coders don't do a good job, or nobody bothered to look at the entry criteria. Of course, too many defects lead you to inspections, not staying focused. So it's a more creative, more less restrictive way of doing the same kind of root cause discussions and brainstorming. What I always say is pick the tool that matches the people that are doing the work. If you've got a lot of creatives, an interrelationship digraph may be more appropriate. If you have a lot of engineers, a root cause analysis, cause and effect diagram might be your better choice. And oh, by the way, it doesn't have to be an actual problem. We can use the same two diagrams for risks and put the potential problem in the head of the cause and effect diagram or the potential problem in the center of the interrelationship Digraph and then look at factors that might cause that to be more or less likely to turn into a problem and have a larger or smaller impact if it does. And we can use that to think about the context of that potential problem and how we might prevent that potential problem. Now, there's many other root cause analysis tools. One of the classics is five whys. What you do with this one is simply just keep asking why? Well, the code doesn't meet coding standard. Why? Well, because the people that are writing the code have not been trained in the coding standard. Why? Because they just got hired three weeks ago and haven't been through the training yet. Why? Because it's not part of the normal orientation program. It's something that's specifically called out by the project managers. And there's no class assignment schedule. Well, why? And we keep going down until we have an actionable root cause. Assuming that there's a single root cause, we can do process flow diagrams. These are simply flow charts or swim lane diagrams. The swim lane is a flowchart where on the left hand side you've got the various roles that have swim lanes across and you put the block or the decision from the flowchart in the swim lane of whoever's responsible for it. But that can help you identify roadblocks or missing steps in your process. Many times if you have a visual to look at and analyze, you can see things that you can't see if you just have written out verbose processes. Fault tree analysis, also event tree analysis. There are two different ways of looking at potentially safety related issues. One of them starts with the safety issue. The boiler blows up and you're looking at all of the conditions down that could have caused that boiler to blow up. The other way, one starts at the events and goes up to the boiler and it's looking at what the probability that each of those events occurs that will result in the explosion. Just simply getting your people to watch for root causes get people to be constantly thinking about this. This is one of the things I love about the agilists, especially the extreme programmers. They talk about Kent Beck in his book. To quote him, practitioners don't just do the work, they think about how to do the work, why they're doing the work, how can they make that work better? They are constantly reflecting. So don't just do post project reviews at the end of the project and look for the problems on the project. Folks, that's just too late. You're not going to help your project at all. Maybe if somebody's paying attention, you might help the next project, but you're not going to help your own. You need to be throughout the process, constantly looking the scrum people at the end of each iteration have a lessons learned session. They call it their retrospective. So that the very next iteration, only two weeks to four weeks later, they're already improving what went the lessons that they learned from the last version. You can just get people together periodically and brainstorm what went well that we need to repeat, what went poorly that we need to fix. And this is what happens in those reflections, in those retrospectives, in those lesson learned sessions, etc. And there's many tools out there in the marketplace. But folks, the intent of this is to identify the root cause, but come up with a solution, a plan, organizational learning. Folks, you did not learn a thing if you don't take action. You just wrote it down and forgot about it. The reason we do root cause analysis is so that we can make things better moving into the future. So root cause analysis is all about continuous, rigorous improvement. There's no such thing as a perfect product. There's no such thing as a perfect process. There's no such thing as a perfect project. Folks. We got to continuously be identifying what we're doing well and repeating it, what we're doing poorly and stopping it with doing good root cause analysis. Any questions, please write them into the question area. I see. We don't have any so far, except a comment from Jamie that says, oh man, now you have to measure to the printer queue length. That's hilarious. Folks. It was a defect. It was a very unhappy customer. We had to go fix it. You need to, when you're writing requirements, make sure that the requirement is inside the boundary of something you have control over and actually write the requirement that way so it can be implemented correctly and quality can build be built in. There's no sense in discovering it once you get to the field. But yes, I appreciate your comment, Jamie. I know it sounds hilarious, but it caused a lot of problems and a lot of angst and a very unhappy customer who put us on a quality hold and wouldn't buy any more product from us until we fixed it for them. That was an easy fix. We went out and bought them a printer that only cost $100 or so, plugged it in. It cost us more than that to deal with the unhappy customer than it did to get them a new printer. But put your questions in on those. And with that questions, ladies and gentlemen.

For the great talk today. Yep. Please put any questions in chat. I've got one for you, Linda. With the more adoption of agile, you kind of touched at this at the end. Do you think teams are getting better at root cause analysis because the agile process kind of forces them to do retrospectives and things on a more frequent cadence?

I think the answer to that question is maybe, and it depends because I see a lot of companies do what Mark Paul calls agile, but they're doing agile but they don't hold regular respect retrospectives. They're doing agile but they don't have a product owner. They're doing agile but they don't refine their stories. So if they're doing it, the answer is yes, and I am seeing a lot more emphasis. And the whole idea of doing iterative incremental development is learning those lessons. Even if they don't do it formally, you can't help but notice your own mistakes and the next time you don't do those mistakes. So the answer for me is yes. I see more and more if they're paying attention, if they're doing agile the way Agile is supposed to be being done. Yes, I'm seeing a lot more organizational learning. I'm seeing a lot less of what Neil Whitten calls professional malpractice. He talks about in his book that we all make mistakes and they end up being defects in our products. Some of us repeat our mistakes because we just. It takes us longer to learn our lessons, but by the time we do the same thing over and over and over again, we got to stop being surprised by it, stop calling it a mistake that led to the defect, and start calling what it really is professional malpractice. So yes, I see the agilist doing a much better job of not only not repeating what they do wrong, but actually repeating what they do right. They have those patterns, they implement those patterns. Release after release, iteration after iteration, they actually put up on the wall, this is what we learned, this is what we're going to continue doing. So people notice and talk about it and think about it and use it. So my answer is yes, if they're doing Agile the way Agile was intended to be. Unfortunately, so many people, a lot of projects I see are using Agile as an excuse to go back to hacking software together and not Agile. The right way is a very rigorous, it's different, but it's still rigorous. In fact, they talk about building quality in and doing it right the first time and it's the only way of being truly agile.

So yes, a question that was just put in chat. So the whole team takes ownership of a defect?

Yes. In an agile environment? Yes. And in fact, if you look at extreme programming, whoever found the defect is probably the best person to work with the author or impair programming to fix it and make sure it gets fixed. They do a lot more improvement by what we call storying around or talking to each other versus just doing the lessons learned session, writing it, all the deltas down, sticking it in a file drawer someplace, never to be seen again. Which is kind of the way we used to do it a lot in Waterfall. And by the way, there's some of my fireworks a few minutes left, so.

If anyone has any last questions, please put them in chat.

If not, let me entertain you a second while people are writing those last few questions down. Yep.

Thank you for your time today. It was a pleasure listening to your talk and I hopefully everyone took something away with this that they can use and hopefully we'll have you back again.

Great, great. Let me know whenever. All right, thank you and goodbye.

Thank you.