Writing Testable Requirements

Hello everyone, My name is Linda Westphal and I will be your presenter today for writing testable requirements. What are requirements? Because today what we're going to be talking about is requirements and making them testable. According to Young, a requirement is a statement that identifies a capability, something the software does, one of its behaviors or a characteristic or quality factor, something the software has to be like user friendly, safe, secure, modifiable, supportable, and so on, of a system in order to have value or utility to a user. Now I expand that always to stakeholder, remembering that there are many people that need our requirements that may not be direct users of our system. There may be requirements for customers, there may be requirements for us as the development community, things like portability, maintainability, supportability. We may be our own stakeholders, even though we don't ever actually quote, unquote, use our system. And then there's people that will never be users of our software, people like government regulators. Many of us are in a regulated industry where the regulators are dictating requirements to our software products and systems, even though they're not users. But it should have some kind of value or utility to a stakeholder. It's a user need, necessary feature, function or attribute of a system that can be sensed from a position external to that system. According to Alan Davis, basically what this is saying is it's defining the system from an external viewpoint. It tells you what the system is going to be doing, not how. So you should be able to do black box testing where you treat the software as a black box, put the inputs in, get the right outputs that come out to verify the requirements of the system. If you have to get in and look at the code or look at the design, then that's part of the how. I like Somerville's, he talks about it being three basic a specification of three basic items. It's a specification that describes how the system should behave, its functional requirements. For example, if we think of a toaster, what does a toaster do? What is its function? To toast bread? But it also talks about the system properties or attributes. Even if our toaster is toasting bread, if every time we touch it it shocks us, we're not happy with the toaster, we need it to be safe. So those would be one of the characteristics or quality factors, one of the attributes. We want it to toast in a reasonable time frame. If it takes 45 minutes to toast our bread, if the throughput is that long, we're not going to be happy either. Even if it provides its function, which is to Toast. So we want it to do it effectively and efficiently. Maybe we need it to be flexible. I like my toast very light brown. My husband likes his toast very dark brown. What he calls brown good toast, I call burnt bread. And then the last one is the exception to that second billet where we talked about it being able to be sensed from the outside. If you really do have constraints on how the developers, the designers and what choices they're going to make as they implement the software, you can have those design constraint type requirements. So it's what the system does, what the system has to be, and what restrictions are there on developing the system. And we need to remember there's many different levels and types of requirements. As people are writing requirements and trying to write good testable requirements, many times there's a confusion because they are thinking of requirements as if they're all one big thing, but they're not. There's different levels, there's different types. Now, I use pay at the gas pump system a lot. Business requirements are actually telling you why we're even building this. What is it doing for the business as far as fulfilling a business opportunity, taking advantage of a business opportunity, or fixing a problem for the business? For example, for the pay at the gas pump system, a business requirement might be to be able to collect payment for gas out at the pump. Now, that's a very high statement. Can we test that? Yes, but there's going to be a whole lot of other requirements underneath that. As we drill down into user acceptance testing and think about testing it from the stakeholder perspective, they're going to be looking at it. Does it meet my requirements as a stakeholder? So if I'm the purchaser of gas, I need to take that one business requirement. It'll break down into. I need to be able to scan my debit or credit card. I need to be able to select which type of gas. I need to be able to pump the amount of gas I want. I need to be able to get a receipt. On the other hand, if you look at it from the perspective of a different stakeholder, the person who is the attendant of the gas station, if you do go inside and pay, they need to be able to pull the amount pumped and the gallons pumped out on the outside gas pumps without having to go outside and look, they want alarms if there's any issues or problems out there, like the paper's jammed for the receipt printer or it's out of paper so that the customers don't come in and yell. They need to be able to replicate a receipt. If the customer comes in and wants a second receipt or something went wrong outside and their receipt got garbage and the owner of the gas station, a completely third stakeholder needs to be able to do the nightly reports. They need to be able to set the price of gas. So different stakeholders will have different stakeholder requirements. And when we talk about usability from that level, we are typically talking about user acceptance test cases where we're proving the functionality to our user community that will drill down into product requirements. For example, a simple requirement of scan my credit card. The software has to be able to know that you even put a credit card in or that you're scanning your credit card. It has to be able to distinguish whether you're reading a magnetic strip or reading a chip, has to know how to parse the information it's getting from the card reader, how to contact the bank, what information to send to the bank, etc. So you got to drill it down to what does the product have to do. So when we're talking system qualification, testing for the software, we're looking at those product level requirements and testing against those other things that will be at the stakeholder level is those business rules. These are defining the rules we have to live by or that the stakeholders have to live by when they use the system. All your government regulations are going to be business rules. Additional business rules would be things like rules that the gas station has, like what credit cards they accept, what's the maximum per transaction? No more than $100 per transaction on a credit card. No more than two uses for the same credit card per day. And again, those are going to drill down into product level requirements. And of course, your customers, your other stakeholders, ask for it to be fast, ask for it to be safe, ask for it to be user friendly. Now, I don't know about you, but I've never had to learn how to program user friendly. You have to dig that down and decide what's user friendly. And oh, by the way, that varies by stakeholder, because what's user friendly for a novice user may make it next to impossible for a power user because it just drives them nuts. For those of you that remember Clippy that used to pop up in Word, he was real cute when you were learning and maybe even useful, but very quickly he became very annoying. On the other hand, what makes it easy and fast to use for a power user may make it overly complex or very hard to learn for a novice. So we have to drill those down into specific product level requirements. Again, User friendly might mean it has a help function or user friendly might be specific quality attributes related to specific user roles. Like the customer of the gas station can read all of the displays on the gas pump from up to 36 inches away in so many luminance of bright light. Don't you love those gas pumps that are dark gray letters on light gray backgrounds that can't be read in the bright sunshine? Always wondered where the programmer thought these things or the person that picked those screens thought these systems were going to be. I'm sure they were tested in a nice dark building someplace. So we're going to drill down to those attributes. It might result in a product attribute for response time that talks about all user commands will be responded to within three seconds even if it's that little moving dot. And we've got to be able to test these things. The product level requirements have to be written in such a way that a tester can actually come up with a yes, no, this passes. This does not pass. And it's the product level that we're mostly going to be talking about in here during this webinar. But we will mention the stakeholder as well. But the stakeholder is the one that's sitting there saying yes, that's user friendly for me. Other things that happen in the product level are those external interface requirements, design constraints like we talked about. Now to test design constraints you're not going to be able to use black box testing. You're actually going to have to peek in to the software and how it's implemented to test those any data requirements and all of these will be specified in one or more requirement specifications. And I don't care if they're in many documents, a single document, post it notes or 3 by 5 cards up on the wall like the agilist do for their user stories. Of course, then they drill it down to a set of acceptance criteria which really define those product level requirements for them. Or in this day and age, many of us put all of our requirements as data entries into a database. So each requirement is an individual data entry. And I love this quote from Fredericks Brooks. The hardest part of building a software system is deciding precisely what to build. No other part of the conceptual work is as difficult as establishing the detailed technical requirements, including all of the interfaces to people, to machines and to other software systems. No other part of the work so cripples the resulting system if it's done wrong. No other part of the work is more difficult to rectify later. So one of the main reasons we want good testable requirements is to be able to test them early and often and make sure we have them right so that we can build that high quality product so that we can make sure that everybody understands them, the designers and developers are interpreting them correctly, the customers understand what they're getting, the testers can test them and make sure they're met long before we deliver the wrong product to the customer. So what are my tips for writing good testable requirements? And this is just an introductory slide. We'll talk about each of these very specifically during this webinar. But we want to write short, direct, action oriented, complete sentences, so we don't want people having to guess what this means or when it happens, those kinds of things. We want to make sure that the requirements are internally consistent with each other. How can you test a system if two different requirements contradict each other? If this one is testable and passes those tests, then this one's going to fail the test. If it's not internally consistent and it needs to be externally consistent, the product level requirements need to be consistent with the stakeholder requirements, which are consistent with the business requirements. Or again, we're going to have testability issues. We want to remove as much ambiguity as possible. Now, folks, stop now. Your requirements are in natural language, which is the way we usually write them. There's always some level of ambiguity and we'll talk about that. But we want to remove as much ambiguity as possible so that all of our stakeholders that know the product, that are working on the product, that are buying the product from us, that are going to be testing it, all have that common understanding of what the requirements meant. We don't care if it's ambiguous. For George here out on the street, who knows nothing about our context, nothing about our product, that's not the level of ambiguity we're trying to remove. We're trying to make sure that everybody who's directly involved understands and that ambiguity has been removed. We want to make sure that the requirements are measurable and we'll talk a lot about that in here. I'll give you some templates, etc. Because if they're not measurable, how can you test to those specific measurements? There's no way you can test quickly because is that two nanoseconds, Is that two minutes? Is that two hours? What's quickly? And different stakeholders may have very different perspectives of that. We want the requirements to be finite and we'll mention talk more about that. And we want to include testers in our requirements. Peer review. Good Testable requirements are identified very early in the life cycle. If they're not testable, they're also not going to be designable and codable. So we want to test early. We want the testers in there thinking about how to write those test cases and how to strategize about testing each requirement very early in the life cycle. And finally, we need to, again, as soon as we have those requirements, define a test strategy and start writing test cases for each of our requirements. As soon as we start doing that, we will find issues and concerns of non testability. And we want to find those early, not just before we start testing. So we want to write those test cases as soon as we possibly can after the requirements have been locked in. But again, at the last reasonable moment, we don't want to waste a whole bunch of time having to rewrite and rewrite the test cases. So we want to drill it down and write those as we are implementing the requirements or getting ready to. And we finally want to make sure that every requirement does trace to a set of test cases so that it is being tested. I mean, there's no better way to prove a requirement is testable than have a good set of measurable, finite, unambiguous, consistent set of test cases to test those requirements with. So let's talk about these short, direct, complete sentences, folks. This is not the great American novel. You're not getting paid by the word. Take everything that you learned in English composition and throw it out. This is good English engineering language. This is noun, verb, whatever. Remove as many adverbs as possible. Look back, search for things like ly words and ize words. We're going to do it quickly. It's going to be user friendly. We're going to optimize the screen. We're going to standardize the process. Words like that, by their very nature are not measurable, not testable, because everybody has a different perspective of and perception of those. Turn the grammar checker on and look for passive voice. Microsoft Word will put a green squiggle under things if you turn the appropriate features on and point passive voice out to you. By its very nature, active voice is much easier for people to understand. Passive voice is where the subject of the sentence is buried somewhere down deep in the sentence. Instead of being up front like George went to the store versus the store was gone to by George. You know, it's much more difficult to interpret. And that's a very simplistic version, but it gives you the right idea. Question all your adjectives. Do they really need to be there. Now, the answer is yes, if they add to the interpretation, if they make the requirement much more specific, much more understandable. But we don't want to add so many adjectives that we get lost in the minutia. So this is a fine balancing act and engineering decision. And a lot of that has to do with do we know the context? I'll talk more about that when we get to ambiguity in a few minutes. So what's a good template for good measurable requirement? First of all, tell everybody when this is going to happen. I love this requirement that says the system shall display the gas price right there. That's not testable. When do I display the gas price? Under what conditions do I display the gas price? Is this the ongoing gas price? Is it updated every time it changes? Is it updated as the gas is flowing? Which gas price am I even talking about? So tell them when, when the gas is pumping, the gas pump shall update the ongoing gas price every tenth of a second to reflect the current amount of gas that has been pumped or to reflect the ongoing price of the gas as it's being pumped. You can also of course, add if statements to that when this condition or event happens, if this state or condition occurs, or the status is, or the status is some specific value, then the subject or noun shall do what? Well, we can actually order these now, folks. Again, watch the words. I love bullets. When this conditioner event happens, this subject or noun shall do this, followed by this, followed by this. I don't need to say when this event or system occurs, this subject or noun shall do this. And then if that event occurs, the subject or noun shall then do this. I'm just adding a lot more words and a lot more words. Of course we can incorporate both of these and say if this is true, then it will do this, then this, then this, else it will do this, then this, then this, else it'll do this, then this. So you can combine these as appropriate, but these are those short action oriented sentences that specifically are measurable because it tells you exactly when these are happening and exactly what is happening. Be as specific as you need to be and tell the order in which things are going to happen. If there is no order, you can have a list if it doesn't matter. But make sure if you do that, you're very specific about what order things are going to happen in. You want consistent requirements, you want them to be externally consistent with their source. If you're testing up at the business need, can we prove to the person the customer that's paying for this, that it meets the business needs, that it takes care of the opportunity that solves the business problem. They're not buying software from us, they're buying a solution from us, and they want us to prove to them that we're meeting that solution. Down at the stakeholder level, we need the acceptance tests to be consistent with, and therefore the stakeholder requirements to be consistent with those business rules, those quality attributes, and with the internal requirements of the product. We also want the requirements to be internally consistent at that product level by ensuring that one requirement does not contradict another requirement. As I've already said, if they are contradictory, they can't be tested. One of them is always going to fail the test. Over here it says to do this. Over here it says to do something else under those conditions instead. So which one is right? We want to state each requirement once and only once in our requirements document and reference it everywhere else. If the same requirement is in multiple features, we want to put it one place and say this second feature also includes requirements 313, 314 and 317. Why? Because actually, first for modifiability, because if I go in and modify the document and that requirement is five places that is being modified, I may only find three out of the five. Now I've got inconsistency, and that's going to mess me up when I test. But also to minimize the amount of wording, etc. It's going to help make sure your document stays consistent. And then finally, use consistent terminology and define it someplace. If I talk about terminating a transaction, I don't talk about it stopping the transaction someplace or ending the transaction someplace. That starts getting very confusing. If I talk about the person that is buying gas, I could talk about them being the driver or the customer or the user or the purchaser. Don't do that. It confuses the development people when they're implementing it, and it confuses the testers when they're trying to test it. Is that all the same person? Is that two different roles in the system? Pick one, define it one place, use it consistently. Of course, just as bad, if not worse, is using the same term to mean two different entities within your system. One place the customer is the person buying gas from you. In another place the customer is the person buying the software from you. So it's two different people, two different roles being referred to by the same title or word. And that gets just as confusing, just as likely to make the software untestable, because they don't know which to believe or which is true. It is also going to make the software more error prone during development because of those ambiguities. Now you all know the nursery rhyme, let's talk ambiguity a little bit. Specifically, you all know the nursery rhyme about Mary had a little lamb. Well if you're writing your requirements down, it's very easy for people to misinterpret your requirements because they don't have the tone of voice, they don't have the emphasis. Just this one little phrase can mean many different things depending on where the emphasis is. The agilist talk about having those face to face discussions. The customers, the product owner, the users, the requirements developer, the requirements people, the developers, the testers get together and do requirements through collaboration and talk to each other. This needs to happen in traditional software as well so that we all understand the meaning of these requirements. So I go back to Mary had a little lamb. We can talk about Mary having a little lamb as opposed to Tom or Dick or Harry. We can talk about Mary had a little lamb but she doesn't have it anymore. Mary had a little lamb as opposed to two or three or four of them. Mary had a little lamb as opposed to a great big fluffy one. And Mary had a little lamb as opposed to a pork chop or hamburger for dinner last night. And I bet that wasn't what you were thinking. You were thinking about the critter, not the dinner. The point is folks, if it's written down, you need to have these discussions, you need to be talking to each other about what you meant by them. This is going to help move ambiguity. One of the big reasons that I believe in having team based peer reviews, having walkthroughs, having formal inspections to discuss the requirements face to face, person to person and make sure you understand the context, understand what you're talking about. And of course when we talk context, let's talk about a knife. All of you picture a knife, right now think about a knife. I've asked you to bring me a knife. Now that's a very vague requirement. What do you mean by a knife? Now let's see how many of you fulfilled the requirement correctly. I am sitting here at dinner, I just dropped my silverware knife on the floor and I've asked you to bring me a knife. Do you know exactly what I mean? And the answer is probably yes because you've got that context, you see. And so how much is enough? When we get to adjectives, the answer is going to be just enough to make it very clear and precise. Without getting so many words in there that the meaning starts getting lost in the flowery language. On the other hand, many of you may have had other interpretations because you did not have the context. Some of you may be chefs, and you were thinking about some kind of butcher knife, carving knife, cooking knife, paring knife. Some of you may be boy scouts and you're thinking about that folding knife, the Swiss army knife. Other of you may be a very violent people, and you're thinking about switchblades or knives as a weapon. I may have had some artists in the crowd that were thinking of a palette knife. I may have had some hunters in the room that are thinking about a bowie knife or a hunting or fishing knife. And I may have even had a surgeon or two that had some kind of surgical knife or a nurse. There's many different knives that could fulfill the requirement of just bring me a knife without the context when we're writing requirements, we do have context. However, we know we're writing for aerospace, we know we're writing for biomed. So don't allow yourself to get so flowery. So many words you lose the context or you overdrive the context and people can't find the short, simple words. So how do we look for ambiguity? First of all, we look for those adverbs like we were talking about words like quickly, user friendly, clearly, easily. Of course, there's other words like maximize, optimize, standardize. But there's also many other keywords. And when I post this out, I actually have a whole page full of words that I've been discovering. Some of them came from several different websites. Some of them are my own words that I've added over time. And I've got a long list of ambiguous requirements words. But what does it mean to say something's flexible? What does it mean to accommodate all users? Now, some of your quality attributes up at the customer level, things like, I want it to be safe. Well, how safe is safe and how safe is safe for whom? And you need to include those kinds of things fast sufficient, keep the pronouns out. Because it is so easy to modify your soft your requirements yet again and lose that contextual reference to those pronouns. George went to the store, he bought a knife. That's obvious to everybody. But if I go in later and modify that requirement and say, george and Simon went to the store, he bought a knife, now I've lost the pronoun reference and it becomes a vague, ambiguous pronoun. So this really is go back to your elementary school. You know, George sees the ball George runs to the ball. Run, George, run. You know, always put the noun in there. Don't use the pronoun and turn on the grammar checker. Look for that passive voice. Make up your own list of words. You can use mine as a starting point, but go from there with words that have been problematic in your requirements documents in the future. An example of both problematic words and making a measurable is something like this. When the monthly report is requested by the user, the report shall be processed quickly. What do we mean by quickly? That's not measurable. How quickly is quick? So we modify it. The monthly report. When the monthly report is requested by the user, the report shall be processed in 30 seconds or within 30 seconds or less. Now it's measurable. We have that 30 seconds and we can run a series of tests making sure that under maximum load, under maximum number of users, all doing the biggest things they can think of, all at the same time. Worst case scenario, it still takes 30 seconds or less. Now, I submit to you, there's still an ambiguous word in that sentence. Have you picked it out? Because we have the measurement of 30 seconds, but can we still measure this? And my answer to you is, this one got us in trouble. This is a real example from my personal history. Why? Because we don't tell them what to measure, just how long the measurement needs to be. What does the word processed mean? Well, to us who are writing the software, the word processed means we've received the command from the user and from the time we get the command, until we have processed the report, done all of the internal work inside our software, posted it to the user's net to print, that's what process meant. So from the time we got the command till the time we posted out, posted it out on their network was 30 seconds. In fact, under worst case condition, it was 15 or less. However the customer interpret process is, it's in my hands and ready to go. I can read the report. And their network was so very slow that it took up to 10 minutes. When you are writing your requirements and thinking about making them measurable, make sure that you're always defining from where to where that you are taking those measurements and always make sure that that was within your scope of control of your system. So this requirement would much better be read. When the monthly report is requested by a user, it will take no more than 30 seconds from the time the request is received until the time the completed report is posted on the network for printing. And all of that is within the control of my system. And Therefore, I can control it. I have no control over the printer, how long it takes the network to print. Now, if this is a completely dedicated printer that I am selling them as part of the system, then I can talk about it being from the time the request is received until the report is printed. Under worst case conditions, of course, I may not be able to get a 50 page report out in 30 seconds or less. So the requirement also has to be feasible for that given printer under those specific circumstances. But my point is here, make sure that everybody knows from where to where that you're measuring and what units of measurement you're doing with specific numeric values. And notice I also say no more than 30 seconds because if I say the report will take 30 seconds to print or I'm going to poll something once every 30 seconds, that's too precise, that's not testable either. Because now we have an issue of what if it takes 30.1 seconds? Do we now fail it because it didn't take exactly 30 seconds? What if it is 30.00001 seconds? That's not 30 seconds either. So how accurately you typically this one's giving you a range no more than 30 seconds. Or it might say it takes 30 seconds plus or minus 10 seconds to do something, that kind of thing. So if it is something, you know, give it a tolerance, give it a range, you don't want to make it so specific that it is unfeasible or cannot be tested in. Then you've got all of your quality attributes and you can come up with specific testable requirements for them as well. I'm not going to talk to all of them, but I am going to mention a few here. For example, usability. It's not testable. I've already mentioned, unless you know, usable for whom? Are we talking ease of learning? Are we talking ease of use? And now we get into specific measurable things. For example, ease of use for a particular person or role. What is the average time or effort it takes to do something? What is the maximum amount of time to learn to do something or to become proficient in a specific action in a specific task under specific known conditions. For example, you're never going to teach a third grader to be able to pump gas because they can't even reach the keyband potentially. Or you know, the specific conditions might be that they can read at a fifth grade level so that they can read the instructions on the gas pump. How long is it going to take an attendant to be able to process a transaction how long is it going to take them on average? Or the maximum amount of time under specific conditions. They've had this amount of training, or it take no more than this amount of training for them to be proficient. Given that they again read at a fifth grade level, they've got at least this much knowledge, they know how to use a basic computer, whatever. Productivity. Now we're talking about the average or minimum quantity of a particular type of task that can be done in a particular time frame. Again, measured from where to where, what's that time frame? From the time they start the task until they finish the last task. And again under specific defined conditions, how quickly the system responds to them. And we've already talked about that kind of thing. You can count their error rates, how many errors they make under certain conditions, or you can talk about how many pages they have to visit to get work done, or how many keystrokes it takes to do work. We want specific measurable requirements with again tolerances or ranges. This is what it takes on average. This is what it takes. Minimum or maximum plus or minus. So many under what conditions and which user reliability is easier to measure. It's very much specific to time. How many. What's the mean time between failure or the maximum number of minutes of failure in a specific time period under specific conditions. Availability is a percentage or again a range of percentages. It will take no more than this percentage during a defined time period that the system will be up and available for doing defined specific types of tasks. We can talk about performance requirements, we can talk about static things like how many, the quantity of printers, maximum, the quantity of users, maximum. The quantity of telephone calls, maximum. Those kinds of things or your more dynamic the throughput, the quantity of work units from where to where during a specific period of time. Are we measuring how many reports can be printed in 30 seconds from the time you start printing them until you finish printing them, or from the time they get put in the queue until you finish printing them. That may make a difference on being able to meet those times. And again response times from the machine or this may be inter machine response times and not response times to the human being. We can also do this with security and talk about specific assets that need to be replaced or excuse me, need to be secure. Specific functions that need to be secure. It's specific data that needs to be protected from people that don't have defined privileges. Specific periods of time that you will wait before you will lock out or place out of service. If there's so many accesses from an external person where the handshaking is incorrect or the logon is incorrect, but we're talking about specific measurable requirements that have the numbers in them. I'm not going to read all of these to you, but we can talk about secure access or how long do we wait before we lock somebody out if there has not been any activity on a specific period of time from where to where? Again, we can do the same thing with safety. What percentage of things do we have to find or detect or prevent? What's the average or maximum number of safety incidents that should be happening within specific time intervals? From where to where under defined conditions? What is a safety risk and how big is that safety risk? What level or variance of safety risk are you allowed to have under defined conditions? And again, and so on you all get so that's talking about making good measurable requirements. Now let's talk about making the requirements finite. We're looking for words like all, most, every, throughout, usually, sometimes, or phrases including but not limited to. I love that one. You might as well just put the word etc. And of course I would tell you, don't make use either one. Change it to to be determined. If you can't come up with a complete list. Now I cannot test etc. Because they can just keep adding things and adding things and adding things over time, etc is non finite. So we're looking for these words that are not again, specific and finite. Now, not all alls are bad. Some alls can be finite if they're within the confines of your system. For example, if I say I will round all monetary calculations to the nearest penny, I know what all my monetary calculations are. Or I'd better if I'm going to go implement that software going to respond to all user commands within three seconds. Again, I know what all the user commands are. Or I better if I'm going to go implement this system. However, I can't accept all credit cards from all banks. That's a bad all because the credit cards are out of my control. They're in the bank's control. And even if I could figure out every credit card in the entire world to implement today, tomorrow the Greater bank of Outer Mongolia comes up with a new Mongo card and all of a sudden I am no longer meeting my requirement. So we want to look for any of these non finite words that are out of again the boundaries or control of our system. I can't say I will respond with an approval to a credit card within 30 seconds because when I send the information to the bank to get that approval, it goes out of the control of my system. What I can say is that if I don't receive a response back within 30 seconds from the bank, I will time out. That's another place where you're looking for finite requirements. Anytime something happens, make sure that there is some kind of obvious limit to it. For example, am I going to sit there and wait forever for a response from the bank or am I going to timeout? What if I ask the user, they've scanned their credit card, they asked me for a PIN number, they get in their car and drive off, they don't ever enter the PIN number, Am I going to sit there and wait forever, locking up that terminal, waiting for PIN number? Or what if they put in their PIN number, it was prompting for gas type, and they drove off? Now is it wide open for the next three hours until somebody else shows up and uses their credit card for them? So for safety reasons, potentially security reasons, we want timeouts on things. We may want timeouts for buzzers or alarms so that they eventually shut themselves off if people aren't responding to them appropriately. So look for things where wait times become infinite, for responses from things that are again outside your control, so that you cannot guarantee that you're getting a response. Now, one of the best ways I know of to make requirements testable is having the testers participate in the peer reviews early and be thinking from that testing perspective. As they're reviewing each requirement, start thinking about how would I test this, how can I test this? Is this even testable? That's their expertise. Let them help the requirements engineer identify those non testable requirements and get them taken care of early before the developers and designers even get them. The testers also have a very vested interest in making these requirements good and testable. Because if they are testable, it's going to make them much easier for the testers to do their job. So they've got a lot of with them, what's in it for me. They also think very differently. They're thinking, how can I break it? So they're going to find things like missing error handling, missing timeouts or requirements that are not measurable, that are too ambiguous, that are not finite, etc. So invite them to the peer reviews and as soon after that as possible, get them defining testing strategies and actually writing the test cases. It's one thing for them to think about it, it's another thing to actually sit down and do it. Now, most of your testing should be able to be done of your requirements at the system level. You should be able to treat the system as a black box or treat the software as a black box and test it. But if it's a design constraint type requirement, you may have to allocate that to an integration level or even a unit level of testing to be tested. So go through your requirements and design a test matrix early on saying each of your requirements. And again, we've got a requirement identifier here. You would not need to write the actual requirement out, but this is the beginning of the test strategy. You can put this into your test plans. But this requirement needs to be tested at the system level through black box demonstration. For example, all the system shall respond to all user commands and data entries within three seconds. We're going to have to do a black box strategy where we again do the worst case conditions of load and capacity and then make sure everything comes back within three seconds. And we actually demonstrate that with a stopwatch and hitting the button on the stopwatch and the computer control at the same time. And then as soon as the response comes back, even if it's that little floating dot or that hourglass, click the stopwatch again or put some way in the software where the software does the measurement. The system shall store all currently active transactions in memory in case storage media fails. Folks, that's telling you how to do it, not what to do. There's many ways of protecting yourself against storage media failing. Rather than storing things in memory, you can have multiple disk drives, etc. That you're storing on. So as soon as I tell you how, I'm going to have to go to those lower level testing and peek in the box and look. The third requirement is also design constraint. It's telling me a specific language. Another example might be to use a specific encryption methodology or specific algorithm. Folks, testing in general, you're going to have to go in and inspect that. You can't test whether a language, a specific language has been used in your software. You're going to have to go look at a sample set of the software. Now this third one, the system shall calculate sales tax at the current tax rates. Anytime you see algorithms, you typically want to test them at both white box and a black box level, or at least an integration in black box level. Have you ever made two mistakes in a calculation and they cancel each other out and you got the right answer? Most of the time, but not all of the time. That's what we're talking about here. We want to do the black box to make sure that nothing's interfering with the calculation, for example, as it is calculating the gas price. Changing that gas price does not change that calculation as it's happening. However, you also want to test at the white box unit level and test each intermediate step of the calculation and make sure each intermediate step is going correctly. For example, I have done spreadsheets in Excel where I put in 3.84, 3.2, but when I add them, the 0.4 plus 0.4, if I only show two digits, if I'm only showing one decimal, even though the entry was 0.84, when I add 0.84 and 0.84, I don't get 1.6, it shows us 1.7. If I'm only doing a two digit display because the 0404 rounds up. So are you rounding before you add or are you adding before you round? And those can mess up your intermediate steps of the calculation. And last but not least, we're looking for this last kind, one that has all of the things we've been talking about here, one or more of them. They're too ambiguous, non finite, whatever, not consistent, so that it's not testable. And we want to find that out as soon as possible, hopefully very quickly after the requirement has been written. One of the things I like about the Agile technique of once they've got that stakeholder level requirement, their story just before they're implementing it, they sit down and talk about the acceptance criteria, which becomes the product level requirements, but also it's the requirements for testing against. So they're in effect writing their test cases at the same time as they're drilling down to their product level requirements or the product level require. The test cases replace the product level requirements, if you want to take it to the extreme. So you're actually going to test each requirement, looking at the, testing the inputs and making sure that it not only works when it's supposed to for the valid ranges of inputs, the valid types of inputs, but it appropriately handles any invalid or incorrect types of inputs or things that are out of range. You want to test the same on the output side. Does it output things that are in appropriate ranges? Can it output something that is so large that it would cause a safety issue, like making the boiler go to too high a temperature that might cause a pressure explosion? You know, are you checking both your inputs and your outputs to make sure they're within acceptable ranges, that they're acceptable types and formats? Are you testing that the data storage is done correctly, that you're not overflowing, that the appropriate security is happening, that you don't have any interoperability lockouts between people you're sharing the data with? And are all your computations being done correctly? We've already talked about that. And one of the last things you can do is trace your requirements at the business level, stakeholder level, product level, down through your implementation to your unit integration system acceptance level commands and review that traceability at the same time you're reviewing those traits test cases, refer them back and make sure that you have adequate test case and coverage for each of those requirements. So just in summary, your testable requirements are short, direct, complete, specific sentences. They are internally consistent with each other and externally consistent with their predecessor documents. The product requirements are consistent with the stakeholder requirements, which are consistent with the business requirements. Good testable requirements are unambiguous, measurable, and finite. To make sure your requirements are testable, get your testers into the peer reviews of those requirements and also have them develop test strategies and write test cases for each requirement as soon as possible. And as those test cases are being peer reviewed, use your traceability matrices to make sure that those test cases adequately test those testable requirements that they truly are testable. And here's the test cases to prove it. This is just some of the references that I used for this. One of the ones that I do want to point out the templates came from. Some of them came from gilb. Some of them are my modifications on ones that have come from GILB over time. But I do want to give him credit for the idea of creating these good testable templates for your requirements. It really helps if you have a starting point or a strawman to start from to help especially people that are newer to writing requirements. Help them write good testable requirements now. Don't necessarily force fit a requirement into the template that gives you a starting point and then wordsmith it to make it good and understandable. I will put my contact information up there. If any of you have questions while you're trying to implement any of this or any questions at all, be sure and contact me. The best way is to reach me is via email or to visit my website. I always encourage people to do that. Thank you all very much for your kind attention.