Security Test Methodology for Testers with Mike Spanbauer

Foreign.

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast, and my goal is to help you succeed with creating automation awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast. So I don't know how many of you know, but I have multiple podcasts. So I have the one you're listening to now, which is the Automation podcast. I also have a podcast on performance insight, reliability engineering called the Test Skill, Performance and Site Reliability Podcast. And I have a weekly news show. It's a 10 minutes news show. It's a audio, it's also a video, it's a LinkedIn newsletter. It's a whole bunch of things and I used to have one that I haven't released one for in a while and that is the Test Guild Security Podcast. And the reason why I stopped producing the security podcast is it wasn't getting as many listens as I thought it should. So I thought of taking one of those episodes and replaying it on the Automation podcast to see how many listens it gets on the automation podcast. Because I'm trying to determine if this is something you find value from. I think as testers we need to be involved in all types of tests and not just functional automation, but also performance automation, performance testing, and also security testing. So this will be an experience to see how many people listen to this and find value out of it. And um, if you don't, if you think it deserves a dedicated podcast, I'd love to know your thoughts. Let me know. Send me an email@joetestguild.com and to check out all our podcasts. All you need to do is head on over to testguild.com podcast, which is plural, and you'll see all the podcasts we currently have running. Also, I'm going to be at StarWest this week, so if you're there, stop by, say hello. I'll be at the karate booth and let me know your thoughts. If you listen to this podcast, if you think you want to hear more about security testing, let me know. Or if you want a dedicated podcast to security started backup, let me know that as well. Appreciate you all. Hope you get a lot of value from this and check it out. Today I'm going to be sharing a session that Mike Spanbauer gave at the last Secure Guild online conference on developing a security test methodology. Mike is a security evangelist for Juniper Security and His work and expertise is a network and security advisory, consultant and product Strategy with over 25 years of experience providing a breadth of perspective as well as providing approaches to solve for operational and governance needs that a lot of organizations face. Before joining Juniper, Mike worked for NSS Labs in their enterprise services and consultant practice where he authored a number of methodologies and worked closely with the testing group to to provide guidance and accurate information to their clients. So as you can tell, Mike really understands not only security but also testing, which makes him a perfect guest for the show. So if you want to know how to create a security testing methodology for your organization, you don't want to miss this episode. Check it out. The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit testguild.comSourceLabs and click on the Exclusive Sponsor section to try it for free today. Check it out.

Hey Mike, welcome to the Guild.

Thanks Joe for having me.

So Mike, let's set the stage. What is a test methodology in your opinion?

What it is is it's a document that provides context of the test and I think most important is to recognize that this is not a document for you, but it's more for those who will consume and rely on the test results, the data on the outside at the end of this testing series. So it's imperative that you include the business or the use case within this methodology to ensure that context is set appropriately, as well as it outlines the assumptions or dependencies. Whether that's environment, whether that's application software, operating systems, cloud infrastructure, tool sets, et cetera. These aspects all go in and it's as much to knock down some of the noise questions so that folks get to meet the test that output itself as it does provide that transparency and trust that the discipline itself followed in the methodology was robust. Again, what we're looking to achieve with the test methodology is that transparency and a little bit of insight into how we created the environment, set up the test series and achieved the outcome so that it's repeatable across both A time. But as much B teams, right. It could be a successor in your role down the line that has to run this test again and a great methodology would allow them to largely pick up what you had done before and achieve the same outcome.

All right, cool. So what do you think should be and should not be in a methodology from your experience?

So what Is in this is the versions, the software, the test bed and the assumptions, the environmental factors. These include, you know, performance characteristics of the environment, what else might have been deployed at the time, production software running as background apps, you know, in the context of web security, you know, understanding which Apache revision and what, what patches are applied, all of these factors are part of that environmental setup, the use case or the validation approach. So what are we looking to prove?

What do we expect this technology to do in the case of an endpoint security product or again some web app firewall, network security? Regardless of what it is that we're assessing, making certain that we are clear in respect to how it'll be deployed and what's expected of it is crucial here, in addition to then the last piece being the test cases themselves buy capability. And so we'll speak to this in a moment a little bit more. But what I mean here is that products often do a great deal more than what we're necessarily expecting or what we'll utilize. So if that's the case, don't try to test everything that a product claims. Rather this is the purpose of the use case. Also focus on the scope that suits your need, your use case and the capabilities required to deliver on that. And so you zero in on what's applicable. And this provides portability between tested products, tested solutions. Now, real quick, as I mentioned, it's also important to recognize what's not in a methodology. So the testing processes themselves, set up, teardown, these things are all implied in the environmental factors, the failures. Right. So the methodology is the reference framework for the test. It's not the things that didn't work to set it up. You know, it's really going to represent what we did use and that's implying successes. So don't worry about the test failures and trust me, we'll spend a moment on this. It was probably where we spent as much energy as anything. Edita says to really validate the tools at work in the context of our client environments and to really deliver review results as well as the last piece is fairly self explanatory, but a methodology is separate from a test result set. So yes, you may only be testing one product or you could be testing a series of products, but those are additional documents, those are separate from the methodology. Again, this is just the legend or the key that sort of sets the stage for everything that follows so that you can run the test time and time again.

So Mike, does everyone actually need a security methodology? It sounds like a little complex, a little cumbersome any thoughts around that? People have asked you that or said that too.

There are options, but there's caveats, and these are questions that you can ask yourself to determine if the scope of the test series or the test case that you're executing.

If it's even moderately complex or it expands past a, you know, a specific period of time which is reasonably discreet. Well, you probably ought to have a methodology for your own sanity and for the benefit of the audiences who will depend on your data, right? So do you plan to run this test series multiple times, or will there be perhaps multiple testers running it? The methodology helps to provide that consistency across the team or across testers in order to achieve that repeatable output. How many audiences are we talking about? Are they, Are they internal audiences or are they external to your organization?

Because it's not likely you're going to be presenting to all of them at the same time and ensuring that the data as well as the guidance is consistent between those presentations. Again, having a methodology will assist to provide both consistency in the message and the content, as well as to provide that grounding and the foundation that allows one to readily tell the story in an incredibly consistent manner without missing something, without leaving something out.

So what does a tester gain out of having a test methodology in place.

For security as well? Are you, the tester, personally available for every results discussion? The reason I suggest this piece is that when it comes to scrutinizing the results data, you know, whether that be performance factor or in the context of security, and largely when you look at, you know, malicious activity, behavioral intent, or you know, exploits, et cetera, there's more than just the data of did it, did it not right? Certainly binary is sometimes the case, often the case for a security efficacy criteria. However, the rest of the attributes such as log detail and such is the color of the conversation. The qualitative piece and the methodology can provide the air cover to a degree to sort of tamper down or to reduce the amount of noise and kind of scrutiny that may come at your test results so that they stand based on the credible merit that they deserve from a well executed test plan without having you necessarily there for defense or for defending the result set each time that somebody asks one of these questions. So I found that that's where a lot of the energy is spent, especially when it comes to disputes, because not everything's always black and white and it's that gray space, grayware or what have you that will create the most, I wouldn't say friction, but certainly contention and most interesting debates. And so with the right setup and definitions of the test case, the explicit expected behavior, expected outcome on a good or a bad result will readily disarm some detractors of the data set of the results and set the stage for, you know, really again, a more impactful consumption of the results that I mean, you know, these results are often quite lengthy to achieve for the whole test plan. Many series we ran were three to six weeks at minimum per product. And in that context, when you look at the aggregate of, say you're running 3, 4, 5, 6 products through a series of tests, in the instances where you cannot run them in perfect parallelism, you're looking at quite a bit of time, some months, in some cases, if not at least hundreds of compute hours where you're running these in parallel. It all comes down to make certain that you have defensible data sets and that it provides that guidance so that you could focus on the meaningful impact of the results versus just defensive the data itself, the tools. This one is I think actually another one of the stickiest points is which version of.

The platform or the tool set, burpsuite, ixia, aspirant, whatever test tools and or products that you're using for your environment, for your series of tasks, knowing which revision was running, what exploits, PCAPs or threat materials are fed in for the purpose of efficacy validation. And so these pieces help at least educate the user and frankly provide some transparency of confidence for the test bed, for the testers in your willingness to share that detail. And it's also a capture once and the value is multiplicative because then you just go read the doc, come back to me if you have questions. The last piece is perhaps, I think, the most notable quote from one of the test architects. And this is that context doesn't transfer. What that means is that without this environmental context, without either the threat kill, tame detail or the tool sets used, then there's any number of questions that could be introduced and that subjects the whole test results set to more intense scrutiny and frankly, just being discarded because of doubt. And that's what we're trying to avoid. The data set is resilient and robust enough to stand against assailants and to really be able to hold its own ground based on merit and based on the discipline executed through the test series. So I think this is where in most instances I'd argue that a methodology, Even a lightweight 1 page, 2 pages, 3 pages, serves the purpose to help you and to help the data set again, really be above judgment Mike, what.

Do you think is the secret to creating an effective security methodology?

What's the secret? What's the formula? It's not a secret, really comes down to domain expertise, testing.

Insights, tools and hard work. Diligence is the key to success, both in a methodology, but also in the full execution of a test plan, a whole test series, and ultimately business resilience. Right? And we look at security resilience and business continuity and these concepts of depending on multiple layers of these technologies in order to effectively run our business and support defense against adversarial action. No matter what size your business is, it's critical. And so we know that hard work is involved. But I mean, I think again, in respect to testing specifically, as a guy who ran a large portion of the organization for quite some time knows, and I had a number of recs open at any given moment, finding both a testing discipline or expertise in addition to the security expertise, domain knowledge of both is incredibly difficult. It's either a commitment to groom for a year or two or you end up having to pay incredibly high figures for capabilities and for folks that can do this.

So, Mike, I know at the Secure Guild you did provide a formula. What's the first part of the formula?

This is the formula that drives what's in a methodology. You start with really getting those use cases. Everything stems from what the product is supposed to do for your organization. What are your requirements, your needs, not what the vendor's selling, not what's written in the data sheet, what do you need it to accomplish. And so this is the top of the tree, if you will, or the start of this flow to begin to develop the series of test cases. So capture that and any factors that are or are not potentially testable this time, it doesn't necessarily have to represent everything that you want to test. And so you can scope the test for either time, reasons or variety of factors accordingly. Just be mindful that it's captured, that that's out of plan, and this is in plan in the context of this test. Again, it's as much for the folks that you will be talking to as those you'll never have a chance to have a conversation with in order to defend your results.

Nice. So what's the second part of the formula next?

The environment, apps, platforms, test tools, versions, software patch revisions, even threats. Right? In the context of capturing threat feeds or pulling in PCAPs, whether you use one of the frameworks or.

Test beds or you've captured your own or you know, obtain those through peer sharing, you know, There's a host of mechanics, we had over a dozen that we used in NSS in order to capture current campaign threats. And so it's imperative though that you have what's being used by the adversaries against the use case, the targets that we're testing for, to both provide credibility to test because there is a currency or a time bound nature. But it's also as much about again representing that this test is a specific scope, it's not a blue ocean, we tested everything, it's just not feasible. There's no product that you could possibly do that with. But also it's not so narrow as to be limited in the value or impact or application of the data.

All right, so that was the second part. What's the third part or third piece.

Of the formula then the capabilities is third and the capabilities is really going to be the bulk of the methodology. And within the capability set, this is test case by test case, the various capabilities of the technology, the solution. And this is again not explicit to security. Right. It's broadly applicable to any technology you're looking to test against. Include the definition of what it is that we're talking about. Performance measures, right. You know, HTTP, you know, or you know.

The packet sizes. HTTPs, what's expected, how large is the frame? Are we allowing jumbos? Whatever's appropriate for the technology to test is captured in a definition so that it's clear what each test case is set up to do. Then.

Walking through what's expected. So a bit of the behavior of capability as to what do I expect, in some cases it's binary, does, does not, in others it's decent. Right. You know, and it's this qualitative grading capture that so that the reader of the results can largely consume your output without again having to come back and hammer you for phone calls. So you know, it's manageable in small orgs, but the larger you get, it becomes rather intimidable. And some of our clients were, you know, multinational hundred thousand plus seat organizations and frankly conversations could occur or need to be occurring in every time zone. So this is the reason for a great methodology because it reduces the number of questions that may come from all quarters and you may never even know where those are going to be coming from. So document.

Well, all right, so now let's bring it all together. What's the last piece of the formula?

You know, what validation details are captured or will be captured. So those don't necessarily need to go into the test report or the results set. But just note that you have captured those for the purpose of validating your result. So this is something that's needed in every instance. But when it comes to, again, that scrutinizing eye, that ability to defend your data, having the details that led you to a conclusion is important. And oftentimes when somebody challenges just because they want to poke, like, look, I've got all the logs, right? I'm happy to send them to you. It's about 12 gig. Where would you like those? If you sort of just share that, yeah, we've got everything. Happy to share. Oftentimes like, okay, they'll take your word for the outcome and the results. At that point they'll have them available. And it's not a bluff, right? Because you'll end up with that one guy who call it and you're like, all right, I'll send it on a drive, right? Give me a couple days to ship you the thumb drive or what have you, make certain you do have that. But capture here in the methodology what you did capture or what you did record during the test series and then finally at the tail, include some of the other aspects of consideration, if necessary or if applicable, Is cost a factor of a specific series of methodology? Deployment, maintenance, deliverables. These are the aspects that I think from a reference methodology that we do a lot of were useful for the private engagements for various companies, various enterprises helping them in their own service architectures and security testing. They weren't as concerned about these facets because they've already got really robust models for most of those operational mechanics. What they were looking for ultimately was integration criteria. Does something work particularly well via salt? What's the extensiveness of the API set? And taking that to its logical conclusion, down the path of is it applicable beyond the immediate test case scope to perhaps a more broad implementation or consumption model?

Where do test tools actually fit into this whole methodology, the tools that you're using?

Let's just say that I have not yet met a test tool that works perfectly. There's always some scenario, some incident, some test schema that causes it to behave a little strangely and not have repeatable results. What I need from the tool side, you know, the tooling itself, it must behave reliably and do so over time so that I'm not questioning the tool. I'm looking solely at product, the technology, the capability under test. And the only way to have confidence the tools behaving as it's supposed to is through validation and potentially a hardening motion. This is not for every tool, but it's because of this that I advise you pick a reasonably robust tool suite and that you learn its quirks, learn it intimately, and understand.

Through repeated testing the rinse and repeat, you'll know where to avoid which tests are reliably solid and thus how to best use it or them for again the purpose of your test schema so this is there's no science to this. This is just time spent with the tools. Every tool has issues. Again, there's no exceptions. Across the dozens and dozens of tools that we used, there's only been a handful of instances where there's not at least one new feature introduced in the process. These vendors are very keen to fix the products if it's a commercial tool or an open source. The community largely already knows about these things and they've documented it and so you don't always have to experience these yourselves. Do a little homework, a little research on the various forums.

On the repos or in Reddit, you'll find an often often case, oftentimes rather a lot of peer commentary and color so that you can avoid these issues that have already been discovered and often experienced by others.

So Mikhail, how would you summarize what the whole purpose is of having a methodology in this whole type of approach that you've given so far?

The purpose at the end of this is again that repeatability, because that's where the trust in the data, in the output, the results lies. And that's one of the keys to a methodology that stands the test of time. But also, as importantly, we'll address those hard questions the critic in the corner and really satisfy them. Like okay, this guy or gal did an enormous amount of work and knows these tools, knows the space, knows the test cases well.

What about peer review? Does peer review come in at any point during this methodology?

You know, it's really just the process of going through requests for comments, not in IEEE or MTF context, but peer review. Really having both your internal organizational peers take a look at or potentially sharing with others that you know are in similar roles in other orgs that are looking at using products in a similar way. It's always great to have an extra set of eyes on a methodology, both for the purpose of sanity checking. Did I miss something? Is there something that I overlooked? Or in addition just simple errors and hindsight's 2020 you'd rather find these things out before you go and conduct a test that runs weeks or longer and ensure that that test is going to be robust according to your orgs use case and needs. You ultimately have editorial rights. And you can not accept that feedback if it makes sense, but communicate to those contributors that there's a specific deadline. Feedback's appreciated. We found that both three to four, five enterprise organizations were always willing to comment and to critique the technology methodologies for, for the security solutions that we ran in addition to those clients dependent on it. And from that, like vanilla or core methodology, we would branch according to specific client needs, but that there was always one master methodology. And that's where version control is critical. Like software, a document must provide that guidance because the test results will be tied to the version of methodology that it was executed against. And that's the reason for this. It's a keyed system. And by interlocking them, then the data and how I did it are really providing that structure so that it stands the test of time. And you may not even be needed to be called in when the test data is reviewed in three months, in six months, what have you. Because it's clear exactly how the test was executed and run.

All right, Mike, I'm sure a lot of people are like, okay, so Mike, so when is this methodology done? When is it good enough?

Well, the answer is never. But the real answer is when is repeatable, accurate? And I think that the bottom line here is that they'll withstand the test time, scrutiny, and it'll pay dividends for you, for your organization. In addition to this holds the technology space a bit accountable. So if it isn't clear, I'm very passionate about accuracy and data as well as transparency and security industry. It's what pulled me to NSS and also what I see in Juniper. At its core, there's a lot of, I think, material value in transparency of data and providing that guidance for an audience so that they can make informed choices themselves. This is the purpose, this is what a methodology serves and what it achieves. And that transparency in the test, in the approach, in the thought process, builds trust not only in data, but in you. This is a career pillar piece, if you will, because I think that as I've mentioned before, testing disciplines, testing as a discipline is difficult in its own right. Adding security to that is, well, it's incredibly unique. And so that combination, that formula, makes you an incredibly valued asset both in your current org, but also throughout the industry. Because the dependence on good security capabilities today is present in orgs of every size, no matter where they're located on the planet. The fact is that in this digitized world, you need great technology that works effectively and does so reliably. And consistently.

This process that I've outlined here works for your own process or product selection, or if you're in a services Org helping support another's choice of product and selection, what have you. It'll also achieve the same and also offer credibility and capabilities as far as how you went about this whole process. And it's not just some random tests like, yeah, we'll do that and maybe do this. And does this sound like any proof of concepts you've attempted in the past?

The process is one that isn't scary. It's not particularly unique or difficult, but it is time consuming and it does require an considerable investment of time and focus.

All right, do you have any recommendations then for folks that are trying to do this themselves?

So my recommendations start with the use cases. Define them well. Be mindful of how long this technology is to last in the environment. It's not just about the capabilities that it's expected to deliver on today. It's also about if this product's expected to live the life of a project for two, three years. Okay, scope, think about what's on the horizon, both from a business perspective and a technology field that may be worth looking at or considering, or at least, you know, keeping on the periphery and stating in the methodology that that's out of scope. We know that that's potentially coming. We're not looking at that today, but it'll be something we could extend as a natural branch of the methodology in a future revision to include in the test scope. Test tools misbehave, plan for course correction, in some cases tossing out a test tool and moving to a different one. The good news is there's a lot of great vendors in the market today that support all manner of tests, test suites and test tools. So finding the right one for you for a specific incident, for a specific opportunity.

Largely there's not dead ends. It's rather the energy spent on validating the tool behaves as claimed. And that it'll suit your needs is where most of your energy was spent. And perhaps most importantly, mind the scope. Don't test everything in the data sheet. Don't try. Right? That's almost certain recipe for failure for significant creep that could extend a project well beyond your ability to execute it. So be clear in the scope, fixate on chunks that are achievable and grow over time, revisioning or versioning the methodology with the iteration as a growth opportunity, however fast you can iterate in cycle.

Handles. And so whether you're in a continuous test process in QA or This is in the context of a discrete technology. This whole concept, the discipline, the approach used here, I've helped guide organizations for years to develop to suit their specific needs. And this has also supported performance measures on processing or storage arrays and other capabilities. So this prospect is not a particularly genius one, but I think it's just rather crystallizing the critical bits, the skeletal elements, and I think the framework that ensures success no matter how it's applied.

Okay, Mike, before we go, is there one piece of actual advice you can give to someone to help them with their security testing methodology efforts?

Step back if you need to. Sometimes you get so close to the problem it can be really hard to see, you know, the path forward. And this is not just to testing, but I find that, you know, more times than not, as we're beating our head against the keyboard trying, you know, to figure out what's, what's misbehaving is it. The tool is, you know, the data just doesn't make sense. Why step back, breathe, and then come at it with a fresh set of eyes or reach out to your peers.

Thank you Mike for security testing Awesomeness for links to everything of value we covered in this episode, head on over to testguild.com a4:18 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Source Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.

Sam.