Integrity & Risk-Based Considerations When Determining Software V&V Sufficiency

Today we are going to talk about integrity and risk based considerations when determining software verification and validation sufficiency. So let's go ahead and get started. As we all know, for any software development project, basically any project at all, you have the fundamental trade offs of cost, schedule and product. When we're doing V and V verification and validation, we are very product focused. We are trying to make sure that that product meets its intended use in the field validation and that it meets all its specified requirements verification so that it's got all the functionality at the appropriate quality levels, that it is secure, that it is safe, that it is reliable, that it is user friendly, all those good quality attributes that we talk about. But it's going to cost us money and take us time to do that. The more verification and validation we do, the more it's going to cost and the longer it's going to take. So we need to be doing these trade offs. When we talk verification and validation sufficiency, we are very much talking about making sure that we have done enough V and V that we can have confidence that our product is high quality, that it meets the functionality and the needs of our users. But we're trying to always minimize the amount of money and the amount of time we spend to get that confidence. So how much confidence is enough confidence? And this is always going to be a basic risk based decision. So we're going to make these trade offs based on risk and based on the integrity level of our product. If we need higher integrity, we're going to spend more and take longer to get there. And for each individual component we're going to look at that risk of that component. So we'll talk about considerations when we're making these trade off decisions throughout our product and project. When we talk about integrity level, we're really talking about what's good enough for the particular business we're in. It's always these trade offs. If we're making a product like a word processor, for example, we've decided to go into competition with Microsoft. If we're that crazy, how good is good enough? Well, we all make comments and in fact there's jokes and everything else about the quality level of Microsoft products. But they are definitely good enough for the industry they're in. I am sure I've met a lot of people that work at Microsoft. There are some brilliant people that work there with enough time and money, different trade offs, they could have ny onto a perfect product. I really believe that with the brain trust they have. But are we willing to spend $3,000 a copy for Word in order to have that high level of integrity. So it's always that balancing act to meet the needs of our customers. Our customers don't want to pay that. They're willing to potentially accept a slightly lower level of quality. And again, I'm not saying we're not building high quality products, but we're always doing these trade offs. Of course, if you go on up the scale and start looking at things that are going to impact people's health, safety and affect their life, things like biomedical devices or airplane navigation systems, then we are starting to be willing to spend $3,000 a copy for Word to make sure that it's nigh onto perfect for the software packages, to make sure it is almost perfect or it's very non defect prone. So we're doing this. So what's good enough software for your particular industry? And this is going to be that balancing act and trade off. And how much you spend on your V and V activities to do your defect correction, to fix those defects, to retest and regression test and repair review after the change has happened to correct those defects is always going to be part of a business trade off based on the integrity level at the highest level of the products that you have. Now just to show you an example of an integrity grading schema that you might use at the highest levels of integrity, a level four, let's say that's the health and safety or it's a product that has to be so secure because it might be national security, or it might be something that might be a target of corporate espionage. Whatever, you may decide that your level of integrity is a level four and you are going to require rigorous levels of all of the various V and V activities. Now this is just an example. I'm not trying to say this is the recommended example. You're going to have to make these decisions for you. But any given company may create different levels of integrity of software. For your highest levels, you may require rigorous implementation of all these V and V activities. As you move down, maybe you're level three level integrity. You don't need quite as much. It might injure somebody, but it's not going to kill anybody. You're not dealing with government secrets or things that might cause severe loss. Then you can deal with tailoring some of your activities. Maybe now you decide instead of full blown peer reviews on your code, you can tailor that down to just doing desk check. At the coding level, you may have your individual engineers doing unit testing on their own code rather than having an independent V and V person doing unit testing, whatever the tailoring is, of your standard processes. As we move on down to the lowest level of integrity, now we can look at planning and using V and V strategies that are lesser required. Maybe you start combining integration and unit testing, maybe much more of your activities are tailorable, or you may have some activities that would not be required at all. So when we talk about the integrity level, we're talking about the integrity level required of your entire system. And you are going to plan your V and V strategies. When you're doing your overall V and V planning, how much V and V is even enough at that level and decide what tailoring is allowable based on your integrity level. Then once we've decided on an overall V and V strategy, we're going to look at the software product that we are building. Even within that product, there may be issues of some components within that product are higher risk, other components may not be quite as high a risk. So you're going to look at risk based V and V for the sufficiency of testing each of your components. And when we talk about risk, we are always thinking about a risk, reward, balance or a risk from the perspective of software quality. On the risk side, we are talking about the probability that there are yet undiscovered defects in your software that would make it all the way out into the field and cause field problems, and then the impact associated with those defects if they did cause that field problem. So we're looking at what is the risk of yet undiscovered problems, yet undiscovered defects in our software causing major field failures. And we've got to balance that against the opportunity that would come from doing additional V and V. What's the cost of that V and V? What's the benefit of that V and V? Simply because we spend more time looking, we spend more money looking, does not mean we're going to find those yet undiscovered defects. Boris Bizer once told me at a conference that every defect you find is going to cost you more and take you longer. Why? Because if it was easier to find, you'd have found it first. So we're looking at, even if we do additional V and V, is it going to have a positive impact on the quality of our software and is it going to find more defects? So we've got to be looking at broadening our perspective on vnv, trying different strategies, different forms of attack, for example, as we said, more broad, different types, going back a slide by doing all of A combination of these different things. In our VNV strategy, we will find more defects. If we spend more time doing just one of these strategies, we, we may not find as many defects. So how can we broaden our horizons to find those yet undiscovered relevant defects? Now, when we are talking risk based and looking at any individual component, let's focus in on the risk side and say what things do we need to test more? What things require more V and V activities like peer reviews? More, more analysis, other activities. So one of the things that we can look at here is the probability indicators that there is a yet undiscovered relevant defect in our software products. What we're looking at here is how risky is a particular component of the software. For example, one of the things that we can do is look at which of our components or, or which of our features or functions within the software have had the most change over time. Every time we touch a piece of code, every time a requirement changes, that causes us to go in and change our design, our code, our test cases. This is an opportunity for the introduction of human error that causes a defect. So if we've had a particular feature that has been very stable over time, the probability we've interjected yet undiscovered defects is low. If the particular component or that particular piece of code has changed over and over and over again over time, these are areas where we may want to focus our V and V attention. What we're trying to do with looking at these probability indicators is, is focus our limited V and V resources because we never have enough time to test it all. We never have enough time to do full blown formal inspections on everything. How do we focus our limited resources, our limited budget, on the things that are most likely to find those yet undiscovered defects while minimizing or resisting overkill on the areas that are not going to be defect prone or where there's less probability that a defect is going to happen or one still exists. If we've already done some V and V activities, the next consideration we're going to look at is past quality. Now what do I mean by that? Of course, if we're looking at legacy code, we can say what features and functions have failed in the field in the past. But even as we are working on brand new software, we can focus in on areas that have had defects. Basically, this is the Prado Principle. 20% of your code is going to have 80% of your defects. I like to think of it as the cockroach effect. I've never had any cockroaches in my house. I'm happy. No bugs here. We can go on. We don't worry about it. If we find one bug in our house, one cockroach, we kill it again. It's an anomaly. We don't worry too much. But by the time we've seen our fourth or fifth or sixth cockroach, we know it's time for some real quote unquote debugging. The same is going to be true with your software. Those 80% of the bugs live in 20% of the code. So by early detection of where the defects are living in your code, you can feed that information into future V and V activities. For example, during the peer review of your requirements, look at which features or functions or which requirements were defective. Where did you find the defects? Which are there particular features or functions that were defective during requirements review? What components during your design were error prone and had a lot of defects in them? By the time you get to code, you're being focused in and you can decide which pieces of code are more risky. Definitely by the time you're into the levels of testing, you know where problems have been in the past. And the more defects that have been there, the higher the probability that there's yet more cockroaches in the woodwork, more defects in that code. I do this even during my system testing activities. I will run all of my previous plan test cases. Maybe I assume that this was going to be a lower risk component based on its pass quality, but as I'm running my system tests again, I'm looking at it from a different perspective than other V and V activities and and maybe I start finding a lot of defects. This might be where it would be ideal based on risk to pull exploratory testing out of your tool belt. I like to always reserve 10 to 20% of my time, at a minimum my test time just in case I'm finding things in a component I did not expect to find time things in so I can pull exploratory testing out and look at that error prone component in even more detail. So if I run my preplanned test cases and everything works, I go on. I'm happy. But if I run my preplanned test cases and I'm starting to find lots of things, then I spend additional time on that feature function, whatever it is I'm testing with exploratory testing to look even further. Of course large complex things are more error prone than small simple things. Your mind can keep 7 things + or -2 in your short term memory, you can keep 7 things+ or -2 in your short term memory. So if things are very large and very complex, you can't think of them as a whole. You have to start subdividing them and paying attention to just this part of the code or just that part of the code, which is the whole idea of information hiding and modularity within your code. But if you have very large components, and again, that's an engineering trade off, you may decide that you need to do more extensive V and V on those, because the human brain can't keep it all in their brain and therefore they are more likely to make mistakes. Of course, the more quality attributes associated with that particular feature, function or component of code, the more constrained it is, the higher the probability yet again that there are going to be defects in it. If it's simple, straightforward functionality that's not tightly constrained, it's going to have less defects. But if it has to have features as well as very secure features, very safe features, very, very user friendly features, the more quality attributes that apply, the more possibility something's gone wrong, the more you're going to want to extensively test to get confidence that not only the functionality works, but that all the quality attributes have been met. So you're going to want to do more V and V. They're going to be riskier components. The operational profile is not so much whether there is that yet undiscovered relevant defect, but whether that defect will cause a field failure. I'm going to use Microsoft as an example again. If there is a defect in the save command, all of us that use Microsoft Word as our word processor are probably going to find that save command defect because we use the save command all the time. However, if that a similar defect is in the mail merge, I don't know if you're like me, but I use mail merge once a year at Christmas time to do my Christmas list. Other than that, not I don't use it very often. So even if there is a defect there, it's not gonna cause a field failure. So we're looking at those defects, trying to make sure we're not missing defects in that highly used part of our software that is going to cause the most difficulty or impact on our users. And we know the skill level, the knowledge level of the people that write the requirements in that area, design the code in that area, implement the code, if they're novices, they're more likely to make mistakes. It's like anything else. The more experienced you are the more, the less likely it is you make a mistake or even if you make one, the more likely it is you will catch your own mistake early and keep it from getting all the way to being a field failure. Now with this one, I don't mean, you know, 40 years of experience in the software industry. I have 40 years of experience but that doesn't mean that I'm not a novice if you ask me to write Java code because I've never written Java code, it's not one of the languages I've worked with. Or if you asked me to do biomed because I've spent my career in other industries but not in the biomed industry, I don't have that domain knowledge and therefore I'm more likely to make mistakes. I'm more likely to have field reported problems. We can also talk about the knowledge and skill level in the current processes. If something has just changed or if I've just moved and transitioned from a waterfall based model to an agile process development, the sheer act of moving to new technology and different technology makes me a novice on those processes. So one way of preventing defects and in fact improving your skills is to do that internal training. This is software as knowledge work. The more knowledge we have, the better we do our work. Another consideration when we are doing this is the technology. If we are doing leading bleeding edge technology, this is brand new stuff no one's ever done before. Again, we don't have the experience with it. We are more likely to be making errors and that turn into defects in our code. If this is the 17th time we've written an energy management and building automation system and we're just adding a few new features to something we've already had, or an accounting system where we're updating the tax tables, whatever well known technology, we're less likely to make mistakes in it. And finally the maturity of our processes themselves. We've got to consider if we are a level one company on the capability maturity model or however you want to specify, if you're doing ad hoc processes, you're more likely to make mistakes. If we have good, well defined processes where we're continuously improving them based on the lessons learned, if we are doing our reflections and retrospectives in agile and feeding those lessons learned back into continuously improving our products, improving the way we do business and we are higher maturity defects are less likely to get into the process, we start mistake proofing our processes and keeping those problems from happening. So that's the probability there's yet undiscovered defects. But if you've taken any classes on risk, you know, probability is just half of the side. Then you got to look at the impact if it does happen. If we have that yet undiscovered defect that makes it all the way out into our operations and we have a failure as a result of that. So we got to look at the impact or the loss associated with that failure. First of all, we've got to look at the cost of that failure to our external users and customers. What did it cost them to have that failure? For example, recently I was working on my book and when I was working on my book, I got real intent on one chapter and of course in the middle of it I had a crash and I lost about an hour's worth of work. Well, that was my field failure cost, an hour's worth of work. On the other hand, if you have something like an airplane navigation system and it goes bad, we may have an issue of loss of life. We're talking two very different costs of a field failure. An example, even within a particular industry. Let's talk your cell phone. If you're on the phone with your best buddy and the line goes dead, what do you do? Well, you hit the redial button. No harm, no foul. Maybe a little the cost or the failure and is a little bit of irritation the 30 seconds it costs you to redial and reconnect. However, if you get a bill from the telephone company that's off by $75 not in your favor, what do you do? Well, if you're me, you're going to get on Muzak and maybe even listen for 45 minutes to Muzak and then deal with it. Maybe you have to make several telephone calls, much more extensive time loss, cost of failure. Of course, if you're on 911 software having an emergency and something fails, you may have loss of life. So which if you're purely thinking about which one you're going to do a full blown formal inspection on, we're talking 911 software. Maybe. The call handling however, is the most used software. So maybe you need to do more extensive VMV on that as well. So we're talking about all of your V and V activities, the reviews, the analysis, maybe even product audits as well as testing. I'll show you some more examples of that and how we might apply this risk based impact in just a minute. Then you've got the corrective action costs. This, this is the cost of actually fixing the defects once they are reported from the field. Examples of this might include how coupled that component is, that has to be fixed to the rest of the world. If you've got to do extensive impact analysis, you've got to fix multiple components. Both the hardware and the software has to get fixed, the documentation has to get updated. That's all more of an extensive cost that than if a single module of code has to be re corrected. Of course we go back to risk based V and V again. Once we do that correction based on how coupled that component is and other risks associated with that part of the software, how extensive do you have to test, retest and regression test? And those are all going to be costs of failure as well. From the corrective action side, then what's the cost of actually doing the maintenance? Can you just put this piece of code out on an Internet someplace and have somebody come grab it and install it? Or is it some kind of firmware chip where you're having to recall boards and correct those boards and reship them so the maintenance costs themselves can be high? We also need to consider customer dissatisfaction. Is this a part of the code? For example, I was extremely dissatisfied when I lost an hour's worth of my work. Having to go back and remember what I changed and what I didn't change and rewrite that. Where if a mail merge doesn't work, maybe I can just hand type the addresses in. It might take me a little longer, but I'm not as aggravated. So are there particular parts of your code or parts of your software that may cause higher levels of dissatisfaction if they don't work appropriately? Of course we're always worried about things like lost marketing opportunities. Even if you're in the gaming software where the level of integrity might not need to be as high as Biomed or nuclear or something like that. If they lose a major marketing opportunity or if they get a bad review in the wrong case, they might lose their entire product or a major part of their market share. What might you get sued about? And of course, all of those of us that are in regulatory industries are always worried about the parts of our software that may cause regulatory non compliance and therefore have a major impact if there are problems in the field. And finally, think about what might cause negative publicity if this particular defect happened. Just ask Ford and Firestone or Jeep Cherokee or Target what negative publicity can have on your particular product line, your particular software, etc. So we've got to be looking, when we're looking at what components deserve more V and V, what components can we do less V and V on, we've got to be paying attention to both the probability and those kinds of indicators as those as considerations as well as the loss or the impact if those problems get all the way to the field. Now these are examples of probability and loss indicators, the kinds of considerations you need to be thinking about. On the other hand, your particular industry may have different ones that you need to be paying attention to. So these are just to give you a starting point and over time you need to pay attention to other indicators as needed and add to these lists or modify these lists over time. So maybe some of the rest of this will help. For for example, what do I mean by tailored? I've worked for companies where their product was so high integrity that if it's level four, everything has to get a full blown inspection. Tailoring it on down might say that the author and moderator get together and make a decision whether a walkthrough or desk check. A lot less rigorous inspection, a peer review might be appropriate for example. On your very low risk stuff, a one person desk check put another pair of eyes on it. An example I use here is my husband. He was also a software engineer and he worked for a company that had one of those, no matter what, full blown inspection on everything. Well, he was working on a little piece of maintenance code and that particular piece of code simply zeroed an array that was used for a report that was not a high risk report. I mean, worst case, you throw the report out, you fix the software, you rerun a report, it was like a monthly summary type report. Nothing that critical decisions were being made on that were time sensitive and the only thing he was doing, the array size on that report got changed from 32 to 64? Well, he's got 30 years of experience in software. What was the probability that if he changed the array size from 32 to 64 in the data declaration statement and in the loop counter, he made a mistake? What's the probability if one other person looked at it, that they wouldn't have caught the mistake? And at this point we've got almost 100% confidence that if there was a mistake there between the experienced developer and the person doing the desk check, that they were going to find it? It's not like it's rocket science to look at two lines of code and make sure 32 is changed to 64. But because he had to do a full blown inspection, he now spent over eight hours of engineering time setting up, finding a moderator, setting up an inspection package, finding two other inspectors having a kickoff meeting, waiting three days because the process Says you have to wait three days before you can have the meeting. People show up to the meeting, somebody's 15 minutes late, you can't start the meeting till everybody's in the room because you've lost some of your capability. So that's, you know, three people sitting there for 15 minutes is another 45 minutes. Then he's got to do the inspection. Of course that takes all of five minutes, but it's interrupted people's flow. They've had to come to the meeting, they have to go back, their work's been interrupted, they have to get back to wherever they were. And then he's got to enter all the information in the database, et cetera, et cetera. And eight hours of engineering time later they have found no defects because there were no defects there to be found to begin with. So this is what I mean by looking at and making good engineering decisions about what's the appropriate level of doing a peer review versus, you know, a one person desk check. If you've got a little bit more risky thing, maybe you let a couple of people look at it and do a desk check of it. Now a desk check is simply individuals working alone, feeding their information back to the author. Walkthroughs are a little bit more formal type of peer review where teams actually get together, but they're not as full blown formal as an inspection. An inspection is a very rigorous process that typically takes more time, it takes more effort and cost. So maybe you can have a small informal walkthrough. The more rigid risky the work product is. Maybe the more you formalize your walkthrough, spend more time preparing, have a bigger team, whatever is appropriate. Once you have a high risk component, maybe you go to a full blown inspection and that's appropriate. Anything again, that goes on the list where it's high cost or high probability, there's yet undiscovered defects if you've got high integrity products. And finally you can make decisions about having multiple peer reviews. So as you're making decisions about what are required in your processes based on the integrity level, you can say we always want to do at least walkthroughs or maybe even full blown inspections on high integrity things. But for level one integrities, we can get away with just desk checks. On the other hand, we may want to do on our really risky stuff, multiple peer reviews. For example, if you're working on high integrity software, the requirements are one, about one of the most risky things you can do, as well as any of the components that are Safety, security, reliability related. So you may want to in the early in the life of those, for example in the requirements, as you are discovering the requirements and eliciting information. For example, I go interview someone based on that interview, I come back, I put together a use case. I could then go do a one on one walkthrough where I walk that customer through the use case I've created. We do an informal walkthrough to discover defects and that person tells me what you know, I forgot a exception here. Or maybe I even ask are there any other exceptions besides these? Are there any other alternatives as we walk step by step through the main success scenario? Then I do other models, I do other use cases, I take them to other customers and having multiple informal peer reviews, desk checks walkthroughs as the requirements are being created and then at the end of the requirements, once I have all of them sit down with the full blown team doing the full blown inspection on everything. Because if you don't get the requirements right, it doesn't matter how long it takes to do everything else. When I'm doing risk based testing, I'm going to focus my testing effort on those high probability that that particular component that makes it all the way into the testing cycles still has yet undiscovered important defects. And what's the impact if those defects do make it past that escape from that V and V activity, especially those later testing activities and would cause a field failure that would be high impact. So again, less testing versus more testing. So our error prone components, I'm sorry, our error prone components, our most used components, our most changed components are all of those things we're going to be thinking about the tighter coupled, the ones that are going to take longer to fix. All of those things are deserving of more testing. Now what do I mean by more testing when I'm looking at implementing risk based testing from a white box perspective, we may look at levels of code coverage. Now that's kind of beyond this particular course to get into that in any detail, but just to discuss it, of course, the least level of code coverage is that I've touched every statement of code. The next level of rigor is that I take every path out of every decision. So that's an if then else type statement. I've taken the if path. I've taken the else path. If it's a case statement. I've tested every single case from a white box perspective. I followed all of the flow of control from a perspective. The next level is to look at the variables those decisions are made on. And look at what's called condition coverage that you've covered every possible condition that those variables could be in. For example, If I had two if statements, one that says if B is greater than 10, and somewhere else that says if B is less than 100, go do something. Now I've got three conditions that B can be. It can be less than 10, it can be greater than or equal to 10, but less than 100, it can be greater than or Equal to 100. So the next level of rigor is to make sure that I've touched that I sample from each of those domains during my testing. Of course, the next level of rigor is to have both decision and condition coverage. And the final level of rigor for code coverage was to do all combinations of all conditions. So if I have one variable like I just talked about with B that has three conditions and another variable that has two conditions, it can be greater than zero or less than zero, then I'm going to combine each of the conditions of B with each of the conditions of the other variable. So that would take me six test cases three times two where it might only take me a couple of test cases to do decision coverage. So we're talking about more and more rigorous. As we get more and more coverage of our code, we may also sample a lot more fault insertion into that code. We can think about all the different ways that the hardware could fail, or all the different ways the user could make a mistake, rather than just focusing on those that are typical. Or most likely, if you have extremely rigorous software, very high integrity requirements, very risky, you would do more fault insertion or more sampling across the potential problems. From a black box perspective, one of the things that we did was spend more time and money doing collaborative test case design rather than having each individual tester design their own test cases where they might forget things or not. Think of all the possibilities of how the user could make a mistake, how the developer could make a mistake, how the hardware could fail and cause software problems, what communication problems there may be with other software packages, etc. Etc. Etc. 1 of the things that we did for our higher risk components was bring a group of people together to brainstorm a list and in fact create a checklist of of all the things that we wanted to test. And we found that by bringing a group together and doing collaborative test case design, we got a more extensive set of test cases, but it cost us more and took us longer. We can look at broader sampling of our input and output domains an example here. Let's say the software is just looking for an input between 1 and 100. Well, we can look at that input domain and break it up into equivalence classes. We have all the numbers between 1 and 100 inclusively that are positive test cases, all the Numbers less than 1, all the numbers greater than 100, which are two negative equivalence classes. So there's three equivalence classes there. Now, how much do you sample from those equivalence classes? Maybe for the lesser risky components you sample each time you run that test case, you just sample around the borders or you just sample one or two items from that domain. If you have more risky components, maybe you sample four or five or six or ten items out of each of those equivalence class domains. Of course, as you rerun regression test and retest that component, you may find you would rerun another set of three samples or six samples or whatever is appropriate. So you're getting broader coverage of your potential input domains. You can do the same thing with your output domains. Look at what the potential outputs are, look at their equivalence classes. Do more sampling from equivalence classes that are associated with higher risk features or higher risk components of the software. You can do more negative testing of your software. For example, again, looking at more extensively, you know what, not just the typical things that could go wrong, but maybe things that would only go wrong under very remote conditions. But if that component fails under any condition, it could cause a health and safety issue. You might decide to go ahead and put test cases together for those very unlikely scenarios where for less risky components you may decide not to put, not to as broadly test all the possible things that go go wrong, just the most likely things that could go wrong. I've already mentioned using that exploratory testing for those higher risk features. One of the things that you can do, as I said, was reserve about 10 to 20% of your testing time. And if you are finding defects in a particular area, either in white box integration or black box testing, go ahead and use that as almost a risk buffer and pull time away from that buffer to go do more extensive testing in the areas where you are finding things. In effect, with exploratory testing, you're designing your test cases as you go. Another consideration would be to do time box testing. This is what the agilistic do. Where they're doing with timebox testing, they are literally saying this is a high risk component. I'm going to give it two hours worth of testing. This is a lower risk component. I'm going to give it a half an hour of testing and what they do is for each of those components they prioritize the list of test cases they've designed. Therefore if they run out of time they've tested the most high priority test cases and their higher priority components are being given more time. So you're time boxing your testing effort ensuring that all the high risk stuff has more extensive testing, that you're spending more time on it, that you're running more test cases against it. The lower priority stuff not so much. So what we're trying to do here is we're trying to avoid overkill by spending a lot of time doing rigorous V and V on components that aren't worthy of that that have very low probability of having those yet undiscovered defects have very low impact if those problems make it all the way to the field and we're spending those valuable VNV cost schedule and resources focusing in on the components that are or that do have that high probability. So that is the end of the webinar. Thank you very much.