Conducting Software Configuration Management Audits

Thank you all for joining me today for my webinar on conducting software configuration management audits. So the configuration management process starts by managing the configuration management function. And then each project or program is going to have a configuration management plan that is going to take the organization level, CM strategy, CM processes and plan how you're going to implement them for that particular project or program. There are four major functions of configuration management. There's the configuration identification, where you identify what you are going to put under control, what level of control each configuration item will be under, when you're going to acquire it, what your baselines are going to be and when you create those baselines, what criteria your configuration items and your baselines need to meet to be brought under control. There's then configuration control, which is all about controlling change to those configuration items to those baselines after you've brought them under control. Configuration status accounting is all of the data collection about your configuration items, your change requests, your baselines, and all of the reporting out of that data. What we're going to be focusing on on this talk is your software configuration audits. We're going to be talking about functional configuration audits, physical configuration audits, as well as in process audits. When configuration management is done well, it's literally the foundation of for the house of good software development, good acquisition, good service. We can probably all cite incidences where configuration management has failed and we've ended up with disasters and development. Something that worked yesterday doesn't work today and we don't know why. Source code that we used to have no longer is around whatever. Good configuration management should be almost invisible. Obviously everybody has to do their part about not allowing unauthorized change to happen, about safeguarding, checking in and out of the tools, etc. The configuration items and good configuration management also supports the release of known products of good status to our customers and users. So configuration management audits provide the objective assurance for in process audits that our configuration management processes are being followed and for the functional and physical configuration audits that the configuration items have been built as required, that they include all the functionality, all of the non functional requirements and and they include the right versions and revisions of all the components. The primary purpose then of these configuration audits is to maintain the integrity of our configuration management items of our configuration management baselines by ensuring those baselines are complete, correct and consistent in relationship to the functional specifications. Those are our requirements, our architecture, our design and our physical specifications which are going to define what versions and revisions of each product are built into each build what needs to be included in each build product, whether it's an executable or a document that's built from multiple chapters, multiple appendices. We're also then going to be approving changes and that they were correctly implemented and verified that unauthorized change was not happening, and that at delivery, the software products are really ready for release, they're of that good known quality. Now, when are configuration management audits conducted? If we look at the various baselines of a traditional waterfall type software development process, you're going to be doing functional and physical configuration audits. At a minimum, when we create that product baseline, just before we get ready to ship the product to the field as a final sanity check, that everything is available, that everything's built correctly, that everything is ready to go. But many times that's just way too late. If we find problems there, we're already out of luck. The risk is just too high. So the recommendation is that you do in process audits throughout the entire development cycle to make sure the product's being built correctly, that configuration management is being done, that we are implementing correct corrections and changes as needed and verifying those as they have been approved by the appropriate authority, that unauthorized change isn't happening, that our baselines are being created with the correct components, etc. So we're going to do these in process audits throughout. But in addition, we may want to do functional and physical configuration audits at the major baselines, when we baseline the system requirements again, to make sure they're complete, to make sure that the appropriate verification and validation has been done, and physical configuration audits to make sure that we've captured all of the requirements, all of the configuration items of that major phase. Again, you can do those again at the allocated baseline where we're taking the system requirements and through the system architecture, allocating them down to the software components. So since we're talking software configuration audits, this is where the software starts coming in. And of course, throughout the development life cycle, we may also want to do physical and functional configuration audits and our various baselines, depending on which development baselines we have decided to use. For example, we had a design baseline that included our software architecture, and we can do a functional and physical configuration audit on that. We had another one that was a code baseline that was built for each piece of code. Now, this was back in the day, so we were doing C code. So that baseline for each module of code included the dot C, the dot dat and the dot H file, as well as all the unit test cases. For that particular module and its detailed design. And we'd run functional and physical configuration audits to make sure that they'd completely implemented all of the requirements, that they had been built appropriately, that the right versions and revisions had been baselined and brought under configuration control. So we can do this throughout the development life cycle. Now, these do not have to be huge, big multi day audits. I've done configuration audits. For example, when we check to make sure the code met coding standard. You know, this is an hour or two where I would go sample 20 pieces of code out of the configuration management library and check to make sure that they had met coding standards. So again, these are not laborious necessarily, but they can be done and should be done as appropriate. And one of the things you're going to define in your configuration management plan is what audits are going to be done, when they're going to be done, and and who's going to be doing them, what you're going to be looking at at each of your configuration management audits. Now, how do you do this in Agile? At a minimum, similar to the waterfall life cycle, just prior to release, you can again perform a physical and functional configuration audit. As you move into the ready to release review and make sure that everything has been appropriately completed. You may also, throughout the development process, come in from the outside and look to make sure that the configuration management processes are being followed. And even at the end of each sprint or each iteration, if you're doing other types of development besides Scrum, you make at the as part of the input into your retrospective or even your review with the users and other stakeholders, you may want to perform intermediate, functional and configure physical configuration audits. So let's look at each of these three types of audits. A functional configuration audit is conducted to verify that the development of the configuration item, whether it be an individual component of code, an individual chapter of a document, or a built component like an executable or a complete manual user's manual or book has been satisfactorily completed that everything supposed to be there is there, that it meets all its performance and other quality attribute requirements that safety, security, usability, expandability, reliability, supportability, modifiability, all of those have been built into the product as well as all of its functional characteristics, and that those have all been verified and validated, and that all the operational and support documents have also been completed and are satisfactory. They were built from the right pieces and parts. This definition is according to the ISO, iec, IEEE System and Software Engineering vocabulary. So one of the main things that I'm going to be doing in this webinar is walking you through checklists for each of these other types of for each of these three types of configuration management audits, just to talk about what we should look at, you know, items on our checklists as well as potential suggestions on how you might gather objective evidence for each of those checklist items. So throughout the development process, at each baseline that you've decided to do functional configuration audits, you can be looking at checklist items that include does the baseline configuration item implement all and only the documented system and software requirements? So you can actually evaluate the requirements to the configuration item forward and backward traceability information for completeness. So you can look at the traceability matrix or, or if you're using some kind of requirements management tool, you can look at the hyperlinks on a sampling basis to ensure that there's no unauthorized functionality has been implemented. You don't have configuration items that don't trace back to requirements and that you've done all of the features and functions that every requirement traces forward to the appropriate configuration items. So that's probably the least rigorous. Just check on a sampling basis, the traceability. You can go a little bit farther and sample a set of requirements. Maybe we'll pick 10 requirements using that traceability matrix, then pull out and review each of the associated baseline configuration items and actually look at the configuration item and make sure it's complete and consistent, that the right versions have been checked in to the tool and put under fault control as appropriate. We can sample a set of improved enhancement requests and review their resolution status. If they're approved, we can then go into the configuration items that were supposed to be changed and make sure that that has been implemented. We can make sure that it's been verified by looking at testing records. And we can sample a set of baseline configuration items and compare them with the previous version. This would be the most rigorous. Actually do a difference between last time and this time and see what changed. What we're looking for here is unauthorized changes. Does every change we have correspond to either a new requirement or an approved change request? If we find any changes in those configuration items for be it a requirement spec, a design spec, an architecture piece of code, user document, whatever, if there's any changes that we cannot trace back to change requests or new requirements, approved change requests or new requirements, then there is a non conformance or non compliance. So we've got that traceability matrix. This is just an example, a TRA Bi directional traceability matrix is considered one of the configuration management tools. Now this one is by no means complete, but every product requirement should trace back to its source. Every product requirement should trace forward into all of the components that that configuration item or that requirement was built into. Now this is not a complete one. It wouldn't have fit on the slide, or at least you wouldn't have been able to read it. It only has unit testing and system test cases. It doesn't talk about integration test cases, user acceptance test cases, maybe even alpha and beta test cases as appropriate. It has just the user manual. There may be other user documentation, an operator's manual, help files. There may be data items, for example databases or data records. There may be user training materials, whatever is appropriate here. And we can, as I said, pull from this. For example, just making sure that every requirement here flows down to the appropriate code and that there are no modules of code that don't trace back to requirements. For example, what else should you do at every time you do a functional configuration audit, no matter which baseline you're doing it at, you want to make sure that all the defects and anomalies reported during the V and V activities of that particular phase of the life cycle or that particular iteration, if you're using Agile, have been adequately resolved and that the or the appropriate waivers and deviations have been attained and that known defects have workarounds that are documented in the release notes. So how are you going to gather objective evidence here? You may review a sample set of approved defects or anomalies. So go in to the configuration status accounting information and sample a set of defect records and look to make sure in that record there's documented evidence that it's been resolved. You can, you can see what it has been resolved in. Hopefully, depending on the sophistication of your configuration status accounting system, the testing record should also be there. We can also sample that same set of defect or anomaly reports and review their resolution status if they got approved. Then we can go actually sample the configuration items and make sure that that defect was actually there. And we can review the actual V and V results. We can go in and re peer review the records, for example, we can look at the peer review records. The we can retest or regression test the test case statuses, whatever. And if we want to really get rigorous, we can actually go in and, and rerun the tests or witness the tester rerunning some of the test cases on a sampling basis. Now, at just the end, at the product and release baseline or the Production baseline. We also want to be looking to make sure that each system or software requirement traces forward into the test cases that verified and validated it. So these would be the system test cases and or the user acceptance test cases where we're going to evaluate the requirements to test traceability. And we're also going to sample a set of requirements and using the traceability information, review the associated test documentation. We can look to see if the test plans were adequate to test it. We can make sure and sample the test cases, look at the requirement and say, do these test cases adequately test that requirement to the level of sufficiency that is necessary based on the level of risk of this particular requirement, the level of integrity needed in this particular product? Now, folks, I've given you several suggestions. Underneath each of these checklist items, you're not going to do each and every one of these. There's different levels of rigor here. Pick the levels of rigor that are appropriate to your particular project, again based on risk, based on the integrity level that you need. I'm not saying do all of this. I'm saying these are choices that you can pick or choose from. The next checklist item that should be done at every product or release baseline is a comprehensive system or software testing. Is it complete, including functional testing, interface testing, the testing of each of your quality requirements. So at the most simple, you can just review the V and V report for adequacy and completeness. Did we satisfactorily complete system testing? Based on the records from system testing, then we can evaluate again the approved test documents to make sure. But now we're looking at the logs, the status of each test case and procedure, and the test metrics. Rather than looking at did we plan to do it? Now we're looking at did we actually do it? Have we completed those tests satisfactorily? And at the most rigorous, we can sample a test set of test cases and evaluate whether or not when we run them, we get the same answers that the testers got when they ran them. Is all the operational support documentation consistent with the requirements and the as built document? As built software. Now we're going to be looking at review minutes from the peer reviews and or defect resolution minutes from the operational and support documents for evidence of consistency. Did we correct everything that we identified in the peer reviews? We're going to evaluate any formal test documents. For example, one of the things we did during system testing is we actually walked through the user's manual, the operator files or the help files and had Test cases and made sure and so what we're looking at here is is that test case coverage adequate to test those operational and support documents, making sure that what the software actually does is what the software is supposed to do. And we can review a sample set of updates to previously delivered documents. So we again go back, sample the documents or sample sections of the document, do a difference between the previous version and and make sure that they have been updated appropriately to match up with the new requirements, any new change requests. So what are these operational and support documents? I've mentioned them briefly, but it may include user or operator manuals, development documents. Even if you ship to your customers your specifications, your requirements, your architecture, etc. Any installation instructions, part of configuration management is making sure that the installation instructions are complete, accurate and actually executable. Even your supporting website, if you have any frequently asked questions, any installation instructions, information about how to contact the help desk. In many cases, this may be one of your primary interfaces to the customer. It needs to be just as high a quality as the product itself. The version description documents need to be correct. They need to list any outstanding defect reports with workarounds. They need to list all of the new features and functions, all of the corrected defects. And that needs to match up with all of the status accounting information. And your training materials are complete, consistent and appropriate. Let's talk physical Configuration audit. A PCA according again to that ISO, iec, IEEE Systems and Software Engineering Spec, a physical configuration audit is conducted to verify that each configuration item as it was built conforms to the technical documentation that defines it. Basically, this is checking to make sure that the right version and revision of each individual component was built into that build. That each coding module that is supposed to be in the executable, the right version and revision of it was built into that executable. That each chapter or appendices or reference that is supposed to be in the user operational documentation is there, but the correct versions and revisions of each of those is there, and that each item corresponds to the information contained in all that configuration status accounting information. So what your records say are true compared to the actual build, the physical build of the products you're going to be shipping. Basically you're making sure all the right parts are in the box before you ship the product to the field for all the baselines. We're going to be looking to make sure that the non conformances and non compliances from our functional configuration audit have been appropriately approved and resolved. Because if there's any missing functions, then you can't have the right versions and revisions of the components until those missing functions, missing issues are corrected. So we're going to review the findings from that functional configuration audit, the associated corrective action records, the associated follow up from that audit, and the verification records on the successful implementation of that corrective action and make sure that they're adequate. Make sure that the corrective action was adequate. We want to make sure all the identified configuration items have actually been baselined. One way of doing that, we can go pull the build report and look at everything that was built into this particular configuration item and then sample those and go look into the configuration libraries, the controlled libraries, and make sure that those configuration items, the right versions and revisions of them, actually exist in those libraries. They've been captured as part of the appropriate baselines. Do all of the baseline configuration items meet their workmanship standards? Now this you have to do as you go along. Nobody's going to worry the day before you're trying to ship the product. Whether the code meets coding standard or the design meets modeling standards, the documentation is in the right format. You've got to be doing this as you go along. So, and I would not even wait till the baseline. At the end of each phase. I may do some of this, as in process audits, for example, I'm not going to wait until all hundred pieces of code are written before I go look at naming conventions and coding standards. I'm going to wait until there's enough code written to sample from. Maybe I wait till 20 pieces of code are written and go look at 10 of them and make sure they meet coding standard, or go look at five of them. If they don't meet coding standard, people will probably go back and fix the ones that don't. If there's only 20 of them and only part of those don't meet coding standard. But if you wait till several hundred pieces of code have been written, it's going to end up being a deviation or waiver because nobody's going to go back and correct that much of the software. We're going to make sure the right version and revision are in each baseline. Now, if we are using a tool, we typically have code lines in that tool, or if you're doing it the old manual way, there's some kind of directory and we're going to make sure that each of those libraries or directories where there's baseline, I. E. This is version one of the product, version two of the product, version three of the product, or build one, build two, build three has the right versions and revisions in it. The last thing we want to do is accidentally build in a previous version of the product of that configuration item into the executable or into the manual, and all of a sudden a defect that we've already fixed reappears. So we can again, on a sampling basis, check those. So has the software been built from the correct components in accordance to its specification? We can gather objective evidence by evaluating the build records against the status accounting information. So the status accounting information says, this is the current version of everything. Does the build record match up with that? Did we build the right version and revision in on a sampling basis? One of the things I always do, if there's any kind of patches or temporary fixes, you need to make sure that those patches or temporary fixes were put in completely and correctly each time you rebuild the software. Typically you only do these, especially if it's a released piece of software, because there's something critical that's happened in the field and you had to patch it or put a temporary fix in it for whatever reason, to correct that critical problem in the field. It's so easy when you put a new patch in to a new build to either forget to put it in or put it in wrong. So at that product release, that product baseline check to make sure just before you ship the product that everything's good here. And you can even sample a set of design elements from the architecture, going all the way back to the specification of the architecture. You decided that all of these components were going to be in this product. Take a sampling of those and compare it with again, the build records to make sure you've actually built all of those components into the actual build of the software. So what we're looking at in our configuration status accounting records is does each software build as we are building it throughout the process? And definitely the ones at the end of the process that are going into the field, were they built with all the correct constituent configuration items? The right versions and revisions of the components were built into the builds. They were tested with, well, with the appropriate test cases, the test procedures, the test scripts. Each individual component was tested. They were tested as they were integrated, and the final build was tested with the appropriate test cases, procedures, and scripts that the specifications align with. Those test cases, procedures and scripts, they align with and are described in all of the specification documents. They align with the requirements, the detailed design, as well as the architecture. And so does the final build that the builds were built correctly using the right versions of the tools, the right versions of the macros, the right libraries, the right version of the development platform, for example, the correct operating system, the correct platform, and that they run on all the correct platforms in all the environments. For example, appropriate configuration testing has been done as part of testing so that you know it runs with all of the different appropriate browsers, it runs on all the appropriate different hardware platforms, etc. And that all the supporting user documentation is there, that it is all complete. So that's that last one. Is the delivered documentation set complete? You can evaluate the master copies of each document against the configuration status accounting information to ensure the correct versions and revisions of the components are there. I do this every time I teach one of my training classes. I always make sure as I'm moving the master copy into our print server library that I've picked up and do a sanity check, that I've got the right version and revision of everything, all the chapters, all the sections, all the figures. I can sample a set of the copied documents ready for shipment and review them against that completed master copy set. Again, I do this. My husband prints all of our training materials in sets of six, because that's how many bins there are on the printer. I always take a nice judgmental sample. I pick the sixth one out of every batch and walk through and make sure the right version, the right revision of each chapter, each appendices are there. This is making sure the replication process worked correctly. Again, all the parts are in the box because it would be very easy to potentially fumble finger hit a five instead of a six. So the sixth copy ended up without a chapter or without an appendices. So I always make sure that those are complete and available. You can evaluate the version and description document against the build records for completeness and you can compare the current build records to the build records for the last release to identify the changed components and make sure that all of the changes for all of the new requirements for all of the approved change requests have been implemented into the documentation as well as into the software. This is one of the things that I find many times the software developers don't do or forget about. They update the software features, but don't go into the help files and update the help file or go into the user documentation and update the user documentation so they start diverging from each other. Again, on a sampling basis, we can check to make sure that they are compatible with each other. Does the actual system delivery media conform to its specification? That is, has it been appropriately labeled? Has it been appropriately stamped or marked so we can evaluate the items on the master media against the required deliverables. Does it have all of its executables, its help file, its data records, whatever's supposed to be on that master media, the correct versions and revisions of that. Sample a set of the copied media to do the same thing, folks, you can make umpteen zillion CDs to send into the field or other versions to send into the field, and you know every one of them is correct right up into the point that the CD driver goes out of calibration. And then you can also make millions of them that can't be read on any machine, but the one that you copied them on actually had that problem once when we worked for the telecom company. We sent out a whole bunch of tapes to our customers with a new release and all the tapes were blank when they got in the field. Another thing that you need to be doing is virus checking those tapes. And it's not a bad idea on a sampling basis to virus check some of that media as well and sample a set of the copied media ready for shipment and review all of the marking all of the labeling against the appropriate requirements and specifications. Do the deliverables for shipment match the list of required deliverables? That is when you're actually putting things in the box. If there's a cd, if there's a user manual, if whatever is supposed to be there is actually there. And then you can sample a set of that ready to release and actually go look at it against the packing list and ensure everything is in the box. If you have any third party software that's included, have you done the. Do the licensing requirements been met so you can look at the build records against the configuration status information identifying any third party components, and then on a sampling basis confirm that we have the adequate number of licenses. Have we met export compliance? Now I add these two here because I've had clients that have actually had these kinds of problems in the past where they've ended up being lawsuits because they weren't paying some of the royalties they were supposed to be paying on third party software. Or they got in trouble because they reused a piece of software and did not realize that it used encryption methodologies that were restricted from being sent outside the United States. So these are. If these apply to you, put them on your checklist. Now, these are just example checklists. Take from this what's appropriate to your products, to your processes. Use the subjective suggestions for collecting objective evidence here, one or more of them, or come up with your own the intent of this webinar is to just give you some ideas to help jumpstart and or benchmark your processes against good practice, but it's only best practice if you adopt and adapt it and make it work for the way you need to be doing your audits. And we'll talk about in process software configuration management audits in general. And in process CM audits need to be value added. It should be worth more to gain the information from these audits than it costs to go do these audits. So don't waste time and resources here. You need a manager that wants this information, that's going to use this information to make decisions. Even if the information is everything's right, no issues, no problems exist. In fact, that's what you want to happen. You want everything to be working according to the requirements, according to the standardized processes and or tailored processes. So in an in process audit you're looking at your CM plans, your processes and or tailored processes, your configuration management systems and make sure they're even adequate to meet your organization's configuration management policies, practices, goals and objectives. If you're required to be a CMMI Level 3 company, are the plans you put in place for each of your projects going to get you to CMMI compliance? If you're required to be NQA1 compliant or whatever, is the what you say you're going to do even going to get you to where you want to be, then you're looking also for are we compliant with what we said we're going to do? So we're looking for are we following our plans? Are we following our processes? Are we implementing our systems? The third thing we're looking for is are they working for us? Are they moving us towards our goals and our objectives? Are we in compliance with our standards and our processes? Are we doing it effectively? Now for internal audits when you're doing these to yourself now if you can also do configuration management audits as second party supplier type audits where you're auditing them to make sure these things are complete at various life cycles stages or just pre production to be willing to accept the product. But for internal first party audits we want to make sure that we're also being efficient in utilizing our resources that we're not doing configuration management in a way that is wasting time, energy and money. Go look for those wastes in our process where we've got too many approvals or our CCB's are at too high a level or whatever and the identification of areas for continuous improvement as you're doing these Audits, whether it be an in process audit like we're talking about here, or even your functional and physical configuration items. Look for groups, teams, whatever that are doing things particularly well, particularly efficiently and identify those to management as positive observations so that those best practices can be shared around the organization. Bill Curtis, in a talk I heard him say once said that if we can take the lowest, worst working group in our company and raise their skill levels, their way of implementing things up to our best and our brightest within our company, it's typically a quantum leap of improvement. So look for those positive observations, things that are people that are doing things particularly well. Checklist items for M process audits include are there defined CM policies and or standards associated with this process and are they adequate to meet the organization's defined objectives? So we're going to perform a document review of our CM policies, the standard processes, whatever is appropriate here and look to see if they meet our defined configuration management objectives and the policies and standards that we've implemented. We may interview key personnel and evaluate their knowledge of the connection between the CM policies and standards and our organizational objectives with how they're implementing the process, what they're actually doing. The second checklist item, are there defined CM project plans associated with with the processes and are they adequate to meet those defined policies and standards? And oh, by the way and processes. So we're going to perform a document review of the CM plan. If your policy says thou shalt run a functional and physical configuration audit as part of creating the product baseline, then your plans should have that in them, the plans for conducting those audits. So we can actually again perform that document review, make sure that everything that's in the standards, the policies and the processes are appropriately addressed in those plans for that project or program. And again, we can interview key personnel on that project, people conducting the configuration management activities, make sure they're aware of the plan, make sure they know what they're supposed to be doing. Are the procedures and or work instructions for the process adequate to implement those defined policies and plans? So number two is do we even exist? Do they meet those defined policies? And now have we implemented them? And we're going to perform a document review of those plans and make sure that they're adequate against those policies. Does each person performing the CM associated tasks within the process have access to those applicable procedures and applicable work instructions? So now as you're interviewing people, not only talk to them about do they know what they're supposed to be doing, but ask them, you know, if I had a Question where would I go find this document or this process or this standard or procedure? Make sure that they have access, that they can pull it up in a quote unquote reasonable manner. And when they do checklist item says 5 checklist item 5 states make sure it's the latest version in revision. So again, on a sampling basis, as you're looking at these, do they have access to them? Is that access to the latest version in revision? Are the entry criteria for each of your CM processes actually verified before the process begins? So before you do a configuration identification, have you met the entry criteria before you do a configuration audit? Folks? Audit should be auditable. Have you met the configure the entry criteria for doing that activity? So you can interview a sample of people performing the tasks and talk to them what an interview question might be something along the lines of what has to be true before you start this process? And then when they answer that, how do you verify that those things are true? Or you can examine some kind of quality record. Maybe it's an entry criteria checklist or maybe it's minutes from the review meeting of those entry criteria. Does the person performing the CM activities have the appropriate education, training, skills or experience? Of course. We can sample a set of personnel and talk to them about their knowledge, their skill, their experience, ask them about training they've received and evaluate answers to the process to the their evaluate their answers against the requirements for training within the process process, we can observe the tasks actually being performed. We can go watch somebody opening up a change request, we can go watch a change control board process happen and determine whether it's actually being performed as specified? Or yet again we can evaluate those quality records. Are the checklists completed? Are the appropriate data being recorded? Are the appropriate minutes there? Did the right person attend the meeting or did they delegate it down to a subordinate at such a low level that the right that they don't have any decision making authority, they got to come back, get an answer and then go back to the team. Does everybody performing the CM tasks comply with the policies, procedures and standards? Again, are the people doing what they're supposed to be doing? We can interview them, we can observe them doing it, we can look at those sampled quality records again to verify that things are being done the way they're supposed to be done? Is the environment, the infrastructure, the tools being utilized adequate to achieve conformity with those? So we can again interview people if there's anything laborious or wasteful about the tool sets, are they being able to collect the right records to get status accounting, can they get answers to their questions out of the tools? For example, can the tester find out what all got fixed in the last release so they know what to test and what doesn't need to be retested because it hasn't been fixed yet or yet again? We can observe the tasks being performed and ask and ask about the environment, or observe the environment, the infrastructure, the tool tools may include, am I accessing the right version and revision of those tools, of the compilers, the linkers, the operating systems and so on? Just like the entry criteria, we can check the exit criteria from each of the processes and make sure they were appropriately verified before we exited that process. Are non conformities and defects appropriately reported? Are they appropriately tracked to closure? We can interview people, we can sample the quality records. Are the appropriate records being kept? Now here we're going to actually have to sample the quality record. And my concern here is also storage and retention. Are they being kept for the appropriate amount of time based on whatever record retention policies and processes you have in place? Are they retrievable, readable, identifiable? I actually went to do a configuration audit once and in process audit and I thought I was being kind. They said they were going to keep the CCB records for the life of the product in the field plus two years and the product, that particular release of the product had been in the field for two years. The project had taken a year before that. So they should have records that could be as much as three years old. So I thought I was being kind. I asked for records from the previous year. The response I got was, you're kidding, aren't you? And I said no. I said, according to your record retention policies, these need to be retrievable. So he took me out of the building, across the parking lot into a warehouse and to these like this two story warehouse with two story tall shelves. And he pointed up at the top of one of the shelves and said, they're in those boxes somewhere. Folks, that's not retrieval. Especially when I said, no, you need to, I'm sampling this one, this one and this one. Go get me those meetings minutes from those dates. And he says, I'm not climbing up there, but you can go look for him if you want. And they should be have backup copies. I went and audited once and the quality record room had had a flood and all of the boxes of quality records were wet and damp and they from the previous day where they had this flood, they were in cardboard boxes stacked up from the floor. Now I took a nice judgmental sample I went straight to the wet boxes and opened them up. Now if they'd been plastic lined and everything was good, fine. But unfortunately they weren't. The paper was deteriorating, the ink was running so I asked them where their backups were to this paper and their answer was basically backups? What backups? So be they quality records, be they problem reports, all of this should be backed up on a periodic basis as well. So that's one thing you may want to audit and check for. Look to see if the processes are actually effective. And here you're going to be looking for evidence of escapes from the CM process, those unauthorized changes, configuration items that weren't baselined as appropriate, inability to recreate a build, lost intellectual capital and so on. Inability to retrieve those quality records like we talked about. So you're looking for things that went wrong because the processes weren't being followed correctly. Thank you all for attending.