When and How Security Testing Should be Done

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with.

Your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. And today it's going to be something we haven't done in a while. Just you and I talking all about security testing. Now, I know what you're saying. You're saying, joe, why do I care about security? I am a tester. Well, I hope you listen all the way to the end to find out why. I think you are missing out on a huge opportunity to learn a new skill that can make you more employable and more valuable and more valuable to your current employee. Having security testing skills within your toolbox. I'm going to give you a bunch of reasons why and some actions you could take to make yourself even more awesome in 2021. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Okay, so here are some practical reasons why you should be interested in expanding your skill set. Also include security. But I think the more skills you have, the better off you're going to be. So I think we should be embracing more skills and I think as we shift things left, more and more companies are doing this. You're going to need more of these skills anyway. So just having them is just going to make you more valuable to your team and you're really going to start be appreciated and seen and as a leader and winner and you'll start getting promotions or even new job opportunities. So that's where I'm coming from in this episode. And just don't. And just don't take my word for it. I actually was speaking to Angie Jones on an APLIT Tools Future of Testing event a few weeks ago and I brought up the topic of security testing and here's what Angie had to say why she thinks having security testing as as part of your skill set is really critical nowadays. Listen in do you have any thoughts on security aspects of testing? I know you said testers need to get involved with newer technologies, but security testing has been around for a while but something I've been hearing more and more about about testers having to learn.

Any thoughts around that, yeah, from what I've seen mostly in the field, is like a separate expert who kind of focuses on this. And much like, I guess, quality coaches and things like this, they kind of, if they're doing it right, embed themselves in the team and kind of help people think about security aspects. All of this goes with shift left as well. Right. It's not just about functional tests, but things like accessibility and security and performance. These things can be done, done much better if they're done much earlier. Right. So if someone is the security expert and they don't wait until we've developed the product, we've done all the functional testing and we say, hey, make sure it's secure at that point. I mean, yeah, you can do it, but it's kind of late. Wouldn't it have been much better to have that security expert involved? That design phase, when you all are designing all of these different components and how they're working together, wouldn't it be nice for someone to say, okay, here are some security considerations that you need to make before you write any code. Now, you have this in mind as you go to the drawing board to build that feature because, yeah, we do need to sharpen our skills and pick.

Up these new areas.

So I agree with Angie Jones 100%. This is actually a point of view I've been hearing more and more about. I don't know if you're aware of. Not only do I have the Test Guild Automation podcast, but I also have one on performance testing email, and a separate one on security testing called the Test Guild Security Podcast. And on that podcast, I actually interviewed Wilson Marr. Now, Wilson Marr is one of the first folks who had a website that I know of dedicated to automation testing and testing in general. I think in 1996 and 98, he had a site and it ran for many years. He covered Windrunner and Lode Runner. So he has a lot of experience with functional testing and performance testing.

But.

But recently I discovered he got into security testing. So I asked him on that podcast interview why he decided to pivot over to security testing. And here's what he said. Another thing that I mentioned is you've been doing all types of testing from at least as far as I know, like the 90s. So new topic that maybe you've been involved with before, but it's the first that I knew about, is you've been involved in functional testing, performance testing, but what got you interested in. To securities testing?

Yeah, because it is the issue that's really bothering us right I mean, because, you know, it was one thing to be slow, another thing did not quite work. It's quite another to get ransomware, you know, notices. Right? So I think that that is really the biggest concern I think a lot of companies have. And so I got kind of dragged into it, you know, to say, hey, you know, you've looked at this, what do you think of, you know, these things? And I would just horrified to see, you know, how some organizations leave big holes in their, their system. Like I was this one outfit, you know, they didn't even have like the TLS set up where, you know, they were like they weren't using a TLS 1.3, which is the latest now. And they just, you know, they just only had like 11 and 1 2. And and so that really is big, huge thing if you ever looked at the concerns that people have about being able to get snooped. And also the more I looked into it, the more I realized that it is, you know, that's where the money is.

Right.

So that's the reason why I kind of pulled into that kind of stuff. And so over time, you know, I've learned a lot about protocols. And so I took the security plus exam a few months ago and was able to pass it. And so that kind of nailed it as far as know what I can do.

So I know Wilson was semi joking, but he is correct. There's a lot of money in the security field. So having these skills just make you more valuable. And it's also proven out through the demand that's going to be needed for folks with skills with security. In fact, if you look at the US job outlook, there's a official job outlook for careers and how they see them in the next 10 years or so. And the need for security analyst skills is projected to grow by 31% between 2019 and 2029, which is much faster than the average for all skills and occupations they have listed. So it's a great field to be in regardless. So even if you want to stay in testing, having security as potty or toolbox is awesome. But if, even if you want to go into a different field, security testing full time as your full job, then you can't lose. So you have the best of both worlds. You're being, you're upping your skills now to make you a better tester and you could take those skills later. And if you want to move from your QA type automation role and go more into a security position, you could do that as well. And it's not just a trend in the us It's a trend worldwide actually. Once again interviewed Owenani Bestman who's based in the uk. He's one of the top recruiters there specializing in security testing and security. So on my security podcast I asked him why did he choose to specialize in security as where he focuses in on filling positions as a recruiter. I kept on hearing this figure being.

Banded out 4 million and the figure.

Was that there will be 4 million.

Unfilled positions in security by 2022.

So I mean I started when I started doing more research and becoming associated with more associations within security, I realized that there really was a skills gap. No, I completely agree with here as well that there is definitely a skills gap and there's even more of a skills gap when it comes to testers that have a security testing mindset or have even basic skills to get started with testing. And a lot of times I think what holds a lot of testers back from learning more and going to events to learn more about security is a lot of the security events that are out there already are for hardcore security experts already. So from my experience, I know a lot of testers are intimidated by having by going to an event like a defcon or a Black Hat, right? Because a lot of those sessions are aimed towards people that already in the field and the security focus, but not necessarily tester focus. So I thought there'd be a need for a online event that is more inclusive or tester friendly for more testers that are kind of beginners, not maybe, maybe not necessarily a hardcore security expert, but maybe want to dip their toe in the waters to learn more about some type of testing that could help them with their security efforts and testing in general. So that's why I created Secure Guild. We're having the second annual online conference dedicated 100% to security testing, which is a tester friendly event taking place October 19th to the 20th. All about security testing from a tester's perspective. We have some of the top thought leaders in the area helping you decide and learn more about what tools to use and techniques to use that you can implement right away and put into place that are actionable within your teams for security testing. So I just want to go over some of the highlights of the event that's going to be occurring, As I said, October 19th to the 20th. So you definitely, if you haven't already registered now for the event, head over to secureguild.com, get a ticket, get a ticket for your team because Like I said, I think this is going to be a skill that is critical in 2021 and beyond. So I highly recommend you get on board earlier rather than later. So let me introduce you to some of the speakers and sessions that are going to be that I put together to help you learn security quicker and faster from a tester's perspective. Now, the first thing I know people usually struggle with in general with testing, say even Autom, is having a web application or environment to test against so that as they learn, they're not worried about affecting any production level application. So we have a few events that's going to help you set up your own environment that will teach you how to hack. The first one is with Royce Davis. Royce Davis is actually the author of the Art of Network Penetration Testing. And he has a really great concept that he went over in the book that he's actually going to share with us in a live session at the conference on how to create a.

Environment where you could teach yourself penetration testing with the hands on walkthrough of an environment he has that's open source called Capsule Corp. Pen Testing Environment. And once again, I've spoken to all these folks on my security podcast and here's what he had to say when I brought up that particular topic and I have to speak with him. I said, oh wow, we have to actually get a session for the folks at Secure Guild on actually how to do this. So here's what Roy said about why this type of approach is really critical for you learning penetration testing. Nice. And I also like in this first chapter you talk about how to set up a lab environment and you have an open source project, I believe that helps with this. Could you just talk a little bit about that for folks that may want to dip their toes in the waters but not compromise in a production environment?

Absolutely, absolutely. So I really wanted to write this book for someone that was conducting an actual penetration test because I felt that there was a bit of a gap between some not so real world training that you might have received to get certification and an actual pen test that the company that you just got a job at would ask you to do. And so I wanted to kind of create and illustrate a set of steps that you could follow to conduct a thorough pen test. But realizing that not everybody has access to do that or maybe the comfort level to do that, I created an open source project that's called the Capsule Corp. Network. And that's just because I'm a huge Dragon Ball Z fan. That's the corny name that I came up with. But what it is, it's a small Windows network that models a lot of the things that I've seen and other people see every day when they do enterprise network penetration testing. So it has active directory, it has database services, it has web applications, it has systems with shared credentials, so on and so forth. All of the things in the book that you see me demonstrating an attack against, you can actually do those attacks against the Capsule Corp network. And you know, you won't break anything or if you do, it's, you can just, you just reset it.

So you definitely don't want to miss Royce's session at Secure Guild on teaching yourself penetration testing. A hands on walkthrough of the Capsule Corp Pen testing environment. Kind of along the same lines, we're also going to have a session by Dale Meredith on creating your own hacking lab, which is basically going to be in a laboratory where you can practice your skills and test new vulnerabilities against an environment that's similar to the one that you will be testing against in production. But it's production network safe, so you won't be able to harm anything. But it's still kind of your environment. So really cool session there as well. I think this is really critical to really be able to get hands on in practice and not have to worry about affecting or creating issues in your production or your real data. So here's Dale talking about once again in my interview with him on the Secure Guild podcast. A little bit why this is important. So one of the things you mentioned and that I really enjoyed about the course is this. Create a lab to hack safely. Could these labs be used to do some what if scenarios? Oh, people could play in almost like a playground.

Yeah, absolutely. And what's beautiful is that you can do this anywhere from, you know, creating the virtual environment or these environments on your local system. Like what I do in mine, I have a laptop with a bazillion terabytes of hard drive, space, it feels like, and tons of ram. And so wherever I go, I can have an experiment. Even when I'm traveling, I'll jump on a plane, read something, and I'll fire up my virtual machines and see if I can do it. But you can also host these systems in the cloud. A lot of companies are doing cloud environments. And so our goal is to duplicate the network infrastructure. Not machine, you know, one to one, machine to machine. But you know, if we're using SQL in our environment, try to fire up an SQL server and try to duplicate how the app is talking on the back end of it, so. So that we can see if there's any vulnerabilities in that test environment, we can make those changes in the production world. I always try to teach this to my students or viewers. And that is, it's a slogan that I've come up with. And that is just because you can doesn't mean you can. Again, we utilize the same tools. Even on your own network, you shouldn't necessarily use these tools. In fact, my courses are really focusing on building up, I call it a hack lab, which are just virtual machines that people can practice their skillset on that environment. We should never be. Even if, again, even if we work for the company itself, we should never be doing anything without getting permission. And when it comes to permission, there's also the aspect of the scope. And what I mean by that is just because you have permission to check out the marketing department's domain or their area doesn't mean that you can then pivot into the sales folks. And we also don't try to prove our prowess by hacking some company's website and then sending them an email and say, hey, look, I just hacked you. There's a lot of federal and local laws and, you know, black hats as we refer to folks that are doing things illegally. And we want to be more white hats, which means we use the same techniques, we just do it with permission, and we report everything that we see.

Very cool. Thank you, Dale. So another topic that's really hot, especially if you're an automation engineer, is AI and automation. But how. But how about AI insecurity? Especially new attacks that hackers may use to try to exploit AI to penetrate your environment and cause an attack. That's what Dr. Arash will be covering and his session at the Shared Secure Guild. Once again, here's my interview with him on the security podcast, talking about what is an AI adversary. And his session at the Shared Secure Guild is going to be attacking AI with adversarial inputs and how to defend against it. So how does shadow AI maybe expose companies to security risk?

So, just like any other type of, like for example, CyberSecurity, all these AI models are also vulnerable to attacks getting hacked or being misused. So when you developed an AI model, it's usually designed to perform a specific task. So you train it on a training data set, and then based on what it's coming across during training, the model learns to make predictions. And then you take that out and you develop the final product that then is run on your Data set during test time, inference after you've deployed your AI model. When you do that, these models are sometimes vulnerable to noise, which means you can add engineered noise to the input and completely fool those predictions that the model was trained to make. And this is the field of adversarial AI. It's possible for a bad actor to look at your AI model that you've trained and developed and deployed and figure out how to change those inputs. So during inference or during deployment, the model is not performing as it's designed to perform without the user noticing. So that's the tricky part. These changes are so small and so easy to make and so hard to detect that it becomes very problematic for a lot of different applications when you think your AI model is working, but it's not really, and it may take you some time to notice that. So you are relying on these predictions. You are listening to your AI model. What you are looking at in terms of the information you're getting back, they are all compromised.

So actually, to see this in action, like I said, we're going to have a session at secure guild with Dr. Arash walking us through an actual example that you can see to get a feel for what this attack actually looks like and how you can prevent it. And if you've been listening to the automation podcast from the beginning or for any amount of time, you know, I love tools. And so we'll actually have some sessions on tooling as well. We're going to have a tooling on fuzzing. And so Cindy Blake from GitLab, and her session is going to be called Bringing Fuzz Testing to the Mainstream. And here's a little conversation I had once again with Cindy on the secure podcast, a little bit about what is fuzzing, if you're not familiar with that, for the folks, I don't know what is fuzzing, because I know GitLab recently acquired two companies. We'd probably have to tell, or we probably have to get into what the differences are between the two companies, because it seems like they both do fuzzing. So what is fuzzing first?

Yeah, so fuzzing finds unknown software defects. So typically when you're doing scanning for security vulnerabilities, it's known problems, right? There's a database of CWE like known vulnerabilities, and you're looking for those things. Fuzzing looks for those things that we don't even know to ask. And so it can find not only security defects, but quality defects as well. And generally there's Two types. There's what the industry has referred to as black box and white box. We're calling coverage guided and protocol fuzzing. And the coverage guided fuzzing is something that is going to find a different set of things than what the protocol fuzzing would find. And so they're very complementary to one another. And we were able to just close on an acquisition of both Fuzzit and Peach that handle those.

Tanya Janka from SheHacksPurple also will be joining us, giving an intro session on Intro to App Security. Here's Tanya talking a little bit about what is app security. So you get an idea for what you're going to get at the live event with her session ON Intro to AppSec.

Basically, application security is all the things you do to make sure that your software is secure and that pen testing or security testing is just one of all the many things under the umbrella. And I figured out, really loved doing the umbrella, if that makes sense. Like helping them create a program to make sure things are secure, reviewing code, like helping them put tools into their pipeline to make sure it's more secure, creating standards and guidance and helping developers just understand what I actually want from them so that when the pen tester comes, they don't really find very much stuff like that. And so application security is the idea like that you are working full time, usually towards helping the developers make sure they're creating secure code and formalizing activities as part of the system system development life cycle. So sometimes that means going to a design meeting and looking over the design and saying, you know what, this part here I'm concerned about, could we brainstorm perhaps a way that's less risky to accomplish the same path. And so for me, it's as a person who just like loves helping, I really liked that better than pen testing.

Where you go and you do security.

Testing really in depth and you try to figure out vulnerabilities and then you often are expected to exploit them and either get trophies or other proof that not only did you find a vulnerability, but that it was exploitable so that a malicious actor could do something bad with it. I personally would rather spend the extra time when you're supposed to be writing the exploit with sitting with the developer and explaining the difference between a waitlist and a blacklist, and how they add a blacklist, like a list of things that aren't allowed, which I got around, versus a wait list, which is just tell me what is allowed and block every other thing and therefore I won't be able to get in and then helping them program that in if they need the help. Do you know what I mean? And then, okay, so we learned this lesson, so can you teach it to the rest of your team or can I come in and talk to the rest of your team about this? Make sure everyone's doing this.

Now we're also going to have Harini joining us who is one of the.

Lead contributors to the tool Talisman. But her session is going to be on vulnerable dependencies, the toxic relation. And you're actually going to be shocked by how many folks actually have vulnerabilities baked into the application because all the different components they use actually to make up their system. So really important topic. Here's Harini, a little bit flavor for her intro that she'll be giving for her session.

Hi, I'm Harini and I work with ThoughtWorks as a capability lead. I've also played the roles of security practice lead and quality analyst and various other roles in delivery.

I'm also a core contributor and maintainer to Talisman, which is an open source tool which prevents you from checking in secrets. And I am also an evangelist of open source in general and I take this opportunity to welcome you all to Secure Guild. And speaking of security, there are a lot of relations that are code hold and we try to protect all of them. We try to secure our data, we try to secure the network compliance, even the physical security. But there's one blind spot that we often miss out on, the underlying platforms and dependencies. It is very often right that we want to build a feature and we look for a library and just use that. But have you ever stopped to think how many vulnerabilities are you inheriting from that very library that you're using? 80 to 90% of modern applications consist of assembled components and containers. The top 10 Docker containers, the most popular ones that are there, each one of them have at least 30 vulnerabilities known. And if you are still wondering about the impact that it can have, remember Shell Shock, Heartbleed, Equifax breach, millions of data that were breached, they were compromised and billions of dollars of losses. But don't worry, there are things that you can do about that. And that's exactly what we'll cover in the session today about the vulnerable dependencies and the toxic relations that your code holds with them and what can you do to protect yourself against it.

Thank you, Hrini. Another fun session we're going to have is learning how to threat model using an interactive board game created by Harjit Sandhu, really cool session. Really enjoyed this session as well. I think you're going to find this really helpful to get your whole team involved with security in a fun way. That's going to make it so it's easy for everyone to pick up these techniques and create safer software. So there's a little flavor for Hajit's opening of his session.

Hi, my name is Harjit Sandhu. I'm an application security engineer at zoopla. I help our engineers deliver secure code securely and today I'm going to show you how with little or no security experience, you can learn how to threat model using an interactive game. Using this game, you can also identify risks and articulate controls within a given system.

Engineers and QAs and architects often see security testing as a very specialized skill set.

Leaving it to pen testers to work in specialized environments to capture the security requirements. Meanwhile, security is one of the biggest risks to organization operability today in terms of operability and financial impact.

So anything we can do to tighten the loop on getting some of the security testing done earlier, capturing some of these requirements, spending less time in the performance penetration test environments would be beneficial to the organization and to the engineers themselves. Also, a good way to visualize this is using a software development lifecycle where on the right hand side we have.

Penetration testing, we have release testing, we have user testing, and on the left hand side we have development and unit testing. And if we can push some of these security testing implementation pieces towards the left, we can start capturing some of these security requirements a lot earlier. And this is the whole premise of threat modeling. Now this is not a talk on threat modeling, but really quickly. Threat modeling is a design time tool that helps us look at any given system and recommend controls given the risks that the design allows us to pick out.

Threat modeling is a good tool to have in our toolbox. We need to use it at the right time and we need the right people to use it accordingly.

We can use threat modeling in a way where we go team to team as a centralized team of experts and threat model all the designs or, or we can educate everyone how to threat model and allow the teams to be autonomous in terms of creating their own threat models and articulating their own controls. And on the one side it's like a scale. On the one side we have the centralized team who have high quality threat models but the inability to scale and then we have fully trained engineering functions which are scaled massively. But maybe the quality of the pen testers is a bit lower because they're not experienced in those areas. So we need to be somewhere in the middle because that allows us a good level of quality for our threat models whilst maintaining a fairly high level of scale. I see threat modeling as an achievement that we can unlock and in unlocking this achievement we get a whole new world of opportunity in terms of capturing security risks earlier, spending less time in our penetration test environments and benefiting our customers because we have more inherently secure systems and configurations. I'm of the thorough belief that every engineer knows how to pen a threat model and can understand the risks in their system and articulate controls given those risks. I mean, that's what they do. They're engineers, they break down problems and then they build them back up again.

I was having a conversation with my friend, it was my colleague at the time, Cam, and we were talking about how could we scale threat modeling across the organization with little effort but maintaining a high level of quality.

And we spoke, we're both avid gamers and we spoke about creating a game potentially to allow the engineers to learn how to threat model without maybe needing to know how. I took this, I obsessed about this, no, managed to come up with a prototype and I played it with them. And this is the game I'm going to show you about today. Now with that, I'd like to introduce you today to my game, Threat Hunters. Threat Hunters is an RPG style board game that allows you to present a system and within that system you take it in turns to attack the system and defend the system.

So you definitely want to register today. Head on over to SecurityGuild.com to check out that in all these other sessions and to wrap up the whole conference. At the end of day two, we have Mike joining us going over how to develop a security test methodology. He'll be joining us for a full live session to really wrap it all together. So when you walk away from this event, you have some actual plans and advice on how to go and implement what you learned at this event so your team can get some value right away. Here's Mike talking about a little bit about why security is so important and a little bit about why he is the expert or he is one of the perfect folks to really wrap it all up to go over how to develop a security testing methodology.

But in regards to developing a security test methodology, the role of me running the enterprise practice NNSS was in order to author the testing methodologies for them. So I didn't leave that in the spoiler notes when we originally, when you originally reached out But I wasn't sure if you were aware that I was the methodology architect and author for most of their testing that they published for quite some time. I think what you do and.

This need to help organizations better equip themselves to make their own choices rather than be led by vendors. Helping them, educating them to choose the best technology based on capabilities is by all means the best protection they can have against buying the wrong product. I firmly believe that. I mean, I know our technology is impressive, our marketing is not as good as some of the other competitors, but we've been getting better. But if the products are put to the paces and we don't fare well, then by all means, I want to make sure the customers know they're buying the right products, the right technologies. And that starts with testing effectively, which is why I was so excited when you ping me like, this is awesome. I'd be happy to do that, you know, as a deeper dive. In fact, one of my lead architects also works here too. But I think we could get a couple other voices even. I don't know if that makes sense, but there is a lot of folks that share the same passion you do about testing. And I'd be happy to support and get a conversation going. I'd be happy to support it. My role is, again, getting out and talking with folks, you know, just. It's to try to cut through the mist and the, you know, the fog that is trying to select the crazy insecurity. Right. There's too much fud. There's too much unicorn dust sprinkled about, you know, it's hard to get through it, in fact. Yeah, I mean, I'm very passionate about helping support, you know, these dialogues and getting folks the right guidance for technology purchase decisions that help them.

So that's a wrap for this episode. Hope you got a lot of value out of it and you realize how important security is. I really hope to see you at this year's Secure Guild event. Head over to secureguild.com and register today. Also, share this with a friend or team member that you think would benefit from secure software. And if you don't know, you also will have the confidence of knowing. I've been running online events since 2017. This is not a COVID 19 type of event where I'm pivoting or, you know, jumping on the bandwagon for online events. This is all we do. We do online events for testers since 2017. So we learned a method that seems to work. And a lot of past guilders seem to really enjoy it as well. Really, really hope to see you there. For links to everything value we covered in this episode and to get the link for secure Guild and everything else, head on over to testguild.com a235 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation and security testing awesomeness. As always, test everything and keep the good. Cheers. Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full.

Show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.