250: Pushing Security Testing Left, Like a Boss with Tanya Janca

My third party component scanning thing was like, tanya, you're still vulnerable. Update. Air query. Pew, pew, pew. Chew. Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of Test. Today we're gonna be test talking all about security testing. So I've been trying to learn more and more about security testing and the name that kept coming up over and over again was Tanya Janka. Tanya is a senior cloud advocate at Microsoft and she really specializes in application security and evangelizing software security. She also has a really cool open source project called OWASP Dev Slop and is passionate about public speaking on security and does a ton of things for the community with her various workshops, blogs, community events. Also, as you'll tell from this interview, she is passionate about security testing and also makes a topic like security seem fun. So if you're like me and you always have on your definition of done security, but no one on your team knows what the heck that means, this episode is for you. Check it out. Also, do you know there's a hacker attack every 39 seconds and the average cost of a data breach in 2020 will exceed $150 million. Also, I found out 300,000 cybersecurity jobs in the US are currently unfulfilled. So if you're a software tester, I think you really should start caring more about security testing and actually adding it to your testing toolbox. And that's why I created Secure Guild, an online conference dedicated 100% to security testing. It's going to be taking place May 20th to the 21st, so make sure to register now@secureguild.com I know I've been doing a lot of conferences. I'm just trying to find areas where I think there are gaps that we can actually fill that gap. And I think testers really can fill that role and help their teams learn more about creating more secure software. So if you're like me and you want to learn more about security testing, head on over to secureguild.com test talks is sponsored by the fantastic folks at Sauce Labs, the Cloud based automated testing platform that eliminates the need to maintain your own selenium grid and test infrastructure. Try it for free today. Visit testtalks.com and click on the sign up now link under the homepage sponsor section.

Hey Tanya, welcome to Test Talks.

Hey, thanks for having me.

It's awesome. To have you on the show. I've actually been following you a while on Twitter and LinkedIn and watching a lot of videos that you have around security testing. And that's actually something I find a lot of testers are lacking and probably a lot of developers. So today, that's why I'd like to talk about all your experience around security testing, owasp, and anything to do with security that you'd like to touch on. But before we get into it, can you just tell us a little bit more about yourself?

I am a software developer who turned into an ethical hacker who then got addicted to public speaking.

I was a software developer a really long time, and then I met an ethical hacker. And I was also like a kind of in the evenings, professional musician. And so was he. And we became friends because we're both in bands. And then he's like, you should become a hacker. You'd be so good. And I was like, no, I don't think I like anything better than coding. He's like, no, you're going to love it. And then he was right.

Yeah. And I work for Microsoft and I have the coolest job ever. I basically get to do security research and then release it for free. And I get to write blog posts and create content. And they let me smash all their things. And they're so nice to me.

Yeah.

I tell my boss, like, I'm surprised I don't have to pay you. And then he's like, that can be arranged.

Yeah, it's good. It's good.

Well, that's awesome because it really shines through when you. When you're talking on your presentations. You really do seem to really enjoy your job, which is rare. So why, why are you so excited about OWASP and security testing or security in general?

I feel that security is sort of like taking care of and protecting people, if that makes any sense. And I was really interested in it and I really wanted to do hacking, but one of the first jobs I got in security was actually doing incident response and just being able to let everyone know everything's okay. Being able to figure out what happened and then block it from happening again. Stopping things partway through, I just, I felt so much pride. It's this wonderful impact that makes sense. I get stressful, but it's like, you know, I made sure, you know, that everything. Like, I helped with the election, the previous election.

I was the, like, I ran all the security. I guess I was sort of the boss of that. And.

Just knowing, like, I helped make sure that the election results were delivered properly, that they're secure, that everyone got a chance to vote. It's. I feel like it's noble work, if that makes sense. And there's lots of awesome jobs out there that are great. But I like that. I feel like I'm performing good and I feel that security is definitely in that range where you're protecting the people around you, your neighbors, people everywhere. And then OWASP is the most fun thing that has happened to me in my career.

I used to be a professional musician forever, and music is very, very competitive. And I found that even though, like, I was popular and I played everywhere and everyone knew who I was, I didn't really feel super accepted into the community. But with owasp, like, literally the first time I went, they're like, we can't wait till you come back. And they. I was like, you know, I've heard about capture the flight contests, and they're like, do you want to run one? I was like, wait, what? And then. And they just kept encouraging me more and more. And then I met Nicole Becke, who I started the Dev Slot project with. And she. She's just this amazingly impressive human being, like, who just wowed me. And then she's like, do you want to start a project with me? I'm like, is she talking to me? Do you know what I mean? Where you're just like, what? Yes. And I just feel like OWASP is like, I know there's lots of communities out there, but I'm not religious. And I know a lot of religious communities can be very, very, very accepting of new people. But I just found OWASP is like, yeah, welcome. Like, sit down, you know, let me get you a cup of tea. Like, please be here. We're so happy to have people help with security. And. Yeah, and they taught me a lot of, if not most of the things I know. So it's pretty awesome.

Awesome. Yeah. That's great. So there's a lot of things I like to dive in with that, that you mentioned. The first one is you mentioned elections. You helped security for an election. How do you plan for security for an election? Or any type of application that's complex like that with a lot of moving pieces. And I guess not everything can be a technical issue. You could also have a. I don't know what you call it, but like a soft target or something that's like a social manipulation as a security hack.

Yes. Social engineering.

Yes, that's it.

So I'm not an expert on social engineering, but the main thing is that you need to create policies and then make sure people actually follow the policies. So for instance, don't put them in a situation where they're likely to break the policy. For instance, if you have a door where everyone has to badge in, it's common for people to just walk behind someone else and it's almost rude if they don't hold the door. So I've worked places where they have this turnstile. So it's just impossible.

There's no way you could do it without clearly breaking the law and the person behind you being like, what are you doing?

There's no mistaking that.

When you put in policies and then safeguards to help people obey those policies, you're less likely to run into social engineering issues. But for software.

I have a bunch of thoughts. But the main thing is making sure security is part of the system development life cycle, that it's formalized as part of it. And if you're doing DevOps or Agile or a very long slow waterfall, all of those like, they follow a methodology and you add security into each part of the methodology. So if you're doing DevOps, they love pipelines, I love pipelines. So add security checks into the pipeline, right? So if you're doing. So some tests can be automated and some things you need a human for, as I'm sure you know, since you do test talks.

Some things you need a creative person to come up with ideas and for some things you can just automate them. And I'm building a little pipeline on this little video show that I have with my open source project with Nicole and Franziska Vandana Nancy, There's a whole bunch of us.

Basically I'm just adding different security tools in and doing basic tests right now so developers could push their code and then it does a bunch of basic tests and then I'm going to get more aggressive as time goes on. But the point is that if you have developers following a process, you want to weave the security into that process. So if you are doing waterfall when you. Well actually no matter what, if you're doing requirements right, you want a security person to make sure they add security requirements. So are you doing a file upload? Okay, so if you're doing a file upload, here's rules for that, because that is an extremely risky activity from a security standpoint. So here's like a checklist of things that we would like you to have and you need to have them because later I'm going to upload malicious files.

I have Ones that should set off alarms. And if they don't set off your alarms, then we have a problem.

And I guess the idea is each thing the developers are doing, you need to figure out how you can be a part of it. And especially if at all possible at the same speed that they are moving.

Yeah, yeah. So it seems like developers have a lot of stuff to do. They have to worry about testing and coding and security. And so having checklists is cool, but is there tools that actually are baked into their IDE that can scan in real time? Hey, this is a security risk. Or flag it as you may want to consider this because this is a known security issue before it actually gets into the code base.

Absolutely.

Cool.

Some frameworks like NET and I think Ruby actually just wrote out a whole bunch of bug classes. They output, encode your output. I know NET does that. I don't know if Ruby does that. I know that Ruby makes it so it's almost impossible to do an SQL injection.

And they've done all of these things to make it so that the developers aren't able to make those mistakes. And then I know that there's products that you can buy that plug into your IDE and will actually tell you as you're coding, like, that doesn't look so good. And then there are static code analysis tools that you can run and everyone in industry has huge varying opinions on them, but they'll parse out your entire app into different pieces and then trace through things and point you to where it thinks there might be security problems. Like, oh, you took this variable in from here, but we don't see input validation. And then you use it over there and we think this seems like a bad idea. Then there's dynamic application security testing things where it's like, pew, pew, pew, pew, and it actually attacks your app as it's running on a web server.

That's cool.

Yeah. And then in between there, another thing which I think is the easiest bang for your buck is to do third party component scanning. When we write an app, 60, 70, 80, even 90% of it is actually libraries, then those things have vulnerabilities in them. For instance, I'm using jQuery in my little open source project. Just very recently a big vulnerability came out in it. My third party component scanning thing was like.

So then, so then I like, I need to update that theoretically this weekend in my free time. Yay. And so that then I'm not vulnerable to that risk. Right. And so when you scan your third party components, like it'll tell you immediately if a new thing has come out or if you've added something and it's a bad idea. And there's tons of those and many of them are free or very reasonably priced when compared to like a static application security testing tool or sast, like a code review code analysis tool. They cost so much money. Oh my gosh. Oh my gosh.

You know, that's a great point. I don't know why I never thought of it, but a lot of our code, even the code I write is consuming third party libraries and there could, there probably is a lot of vulnerabilities there. So that's a great tip. So like you mentioned, there are tools then that can scan these third party libraries to confirm that they're okay before you actually consume them in your build.

Yeah, so you can put it. So you can just scan it from your desktop. You can do it. So it will. If you have an open. Actually just if you use GitHub or I'm not sure all the other ones that are available, but they're like the other code. So I basically just use Microsoft products or GitHub. So those are the ones I'm familiar with. Sorry. But for instance, I have an open source project, so all my code is visible in GitHub. And then there's this product called Snyk. And if you have an open source project, it scans it for free once a week and it sends me a little report with their little picture of a dog and it's like tiny up are still vulnerable. Update air query. Yes sir, I know.

And then you can also put it as part of your pipeline as well. So I like to use a bunch of different products because they look for different things and products are made by humans and human make mistakes and errors and miss things. And so if you scan with multiple products. So I use dependency check and Snyk. So I like, I strongly suggest if people have an open source project like setting up Snyk, took six minutes for it to scan my code repository automatically once a week and send me a report. Like six minutes. I was like, I'm just gonna try it out and see so I can prepare for my show. And then I was like, oh my God, I did it and it worked. That was ridiculous. I have to think of something else to do with the other 54 minutes in my show. It's not a bad problem to have, right?

Absolutely. So I guess like you mentioned a lot of times you can actually put this into your CI CD systems. So do you always recommend that when someone checks encoded always runs these security scans or is it like set every week or. A lot of people complain about when they have too many checks in their CI system. It slows down the build process. But I would think this would be critical. Any advice on how to make it so it's not over obtrusive?

I guess I totally have advice on this. Okay, so I have this crazy idea that we should put critical types of scans in the CICD pipeline that go out to publish, but then you can make another like a loop, sort of like a circle. And you put every hardcore security check that's ever happened and then it just goes back to dev and stops. Or it's just like a dead end and it just stops.

And developers could run this on like Friday afternoon or something and it just goes off on its own and runs its stuff. And then the security team can look at all those results and then they can find mediums and lows and other things that are still things you want to fix, but you don't need to break the build over. For instance, for my third party components, I think I allow myself to have mediums and lows, but no highs. And then I run it through SSL Labs, which is another thing that people should consider doing, which checks that you're using, like that you're using the right security headers and also that you are using a certificate, your certificate hasn't expired and all of that. And I need to have an A or an A plus from them or I'm not going to go out. But I am a security professional who is obsessed with security, so you could totally accept a B from SSL Labs. I happen to know people all over the place have D's and F's from them. So if you set yourself like I would strongly suggest setting your breaking of build really low to begin with and then slowly work your way up rather than the other way around. And if you're going to do static code analysis, for instance, just look for injection and cross site scripting and don't look for every other thing. Put that in the big security loop. Because static code analysis literally will run for two straight days. It's insane. So just look for one specific thing and then maybe a week later, once all of those things are fixed, look for another specific thing. If you run like a full static code analysis, all the developers will come to your desk with like pitchforks and be like, our CI CD is broken and it's your fault. And it will be your fault.

Yeah. And for dynamic application Security testing. I was actually just setting this up this morning with this guy named Simon Bennett who created. He's the project leader for the OWASP Zap tool, which is a thing I think all QA people should totally know how to use. And it does security testing, like dynamic security testing. So you point it at your dev server and you kind of go, pew, pew, pew, pew. It will automate a whole bunch of security tests for you. And him and I were trying to make it so it wasn't too intense. So my CI CD pipeline doesn't run forever because that's not acceptable. But again, if you do like a full security loop on the side, that does not go out to prod, that does not annoy the crap out of developers that they don't have to wait for. It's like, please, just humor me and let me run your code through this. And then I'm going to create a backlog for you of things that we can put into some sprints.

I was watching one of your sessions online and you mentioned something that I thought only applied to automated tests where you get a false positive. So are there false positives with security testing, with these scans and these reports? And how do you get it so your developers actually trust where you're telling them, hey, this is a real issue and not some rabbit hole they're going to be going down? That isn't a real issue.

I am so glad you brought this up. Okay, so security tests have a lot of false positives and they have false negatives. So for third party components, they're pretty good because it's kind of true or false, right? Like you are using jQuery 1.1.13 or you're not, right? And either there is a CVE, a common vulnerability enumerator like record for a vulnerability that is in that or there isn't. So those are quite. There's few low, there's few false positives with those. But with static code analysis, it's insane. It's like 95% false positives. And I know what you're thinking. You're like, that's awful. Why would anyone use it? Here's why. So if you do a code review of 20,000 lines, you have to read 20,000 lines of code. If you run stat code analysis on it, it might say, hey, I think I found 200 things. And then you read those 200 things takes a lot less time than reading 20,000 things. And then you find, oh, there's only 10 things. Thanks.

So some people are like, oh, static Code analysis, that really sucks. But if you look at it like that, instead of having to read 20,000 lines of code, you only have to read 200. It does seem pretty good. And it can be really quite good for certain specific things. Like it can be fairly accurate with cross site scripting or fairly accurate with some injection things. Like it's like, hey, you didn't validate that input at all. That's probably bad. It's definitely bad. And then for dynamic testing, I would say that it only finds around 30% of the things that are wrong by itself, but that they're probably 80% correct. So a thing you can do with all of these tools is you can tune them. So you should have a security person first run it manually themselves and look at it and say, okay, so this is a real one, this is a false one, this is a real one, this is a false one. And then they should tune out the false ones. So an example of a false positive that I can think of offhand is I used to use IBM Appscan, which costs more than my car, and you do not need to know like anything to run it. Like, it is really smart, it knows a lot of stuff and it will just automatically do everything you've ever dreamed of, but it tests all sorts of things and it might test things you don't want. So for cross site request forgery, which is a vulnerability that OWASP talks about a fair bit, it's a vulnerability where like, imagine you're logged into Amazon and you've been Christmas shopping for everyone and you're already logged in. And then you receive an email with a link to something, you click on it and then it tries to execute a purchase against Amazon. And because you're LinkedIn or you're logged in, it goes and it buys it and it doesn't check that you're you. So Amazon is absolutely not vulnerable to this because otherwise, like, I'd have a lot of new pants. But.

Like they know what they're doing, they're, they're, I'm sure that they've had people already teach them that lesson. But the way that we verify and make sure this doesn't happen is when you go to do a transaction, like a purchase or deleting your account, changing your password, anything like that, the application, unbeknownst to users, what they do is they revalidate that you're you. So they ask you for a captcha, or if you're changing your password, they ask you to enter your password in again. Or most of the time, what they do is they pass the secret token back and forth. So when this person clicks on a malicious link and then it tries to go purchase something, it's not going to work because there's no token being sent sense with it. And it's like, that's not you. So IBM AppScan checks for this token, but it also checks to make sure the referrer header is from the page that you're on. And again, if you clicked a link in an email, your referrer header would be wrong. But you can spoof referrer headers. So all of our developers are like, but you can spoof refer headers. And we're passing the token and the token works and that's the real thing we should do. So they're like, we're not going to check the refer header because anyone that is smart enough to launch a CSRF attack is smart enough to spoof the referrer header with one extra line of code on top of this huge other giant malicious act that they've done. I'm like, I agree. So we decided to accept the risk and decide not to tell all the developers. They don't have to do that. And then I just went into Appscan and just unchecked that box. So there's different tests. I'm like, just don't do that test. Stop showing me that. And so with dependency check, you can suppress the results and say just like, no, that's not a real result. Like, that doesn't apply to us, or that doesn't matter. And tuning your tools is really important and it's the security team's job to do that. They might need your help, but they definitely should be making sure they don't send false positives because then developers don't invite us to parties and that's no good.

I've been on security teams where the developers, like, they hate us, they don't dislike us, they hate us. And it took a long time for me to trust. Right. And sending them a bunch of false positives just ruins it again, no doubt. Yeah. So thank you for asking that question. That was a great question.

Thank you. So, Tanya, you also mentioned that you're a ethical hacker. So what does that mean? Does that mean someone may have an application, you're going to exploit it, you tell them ahead of time and then you just go hit it and see if you can get in or have a breach found?

It means that if someone pays me and gives me permission and they have the authority to give me the permission, then I will do security testing. On their software or their systems or their networks, but mostly just, mostly just software.

And usually it involves me doing a little bit of threat modeling. Like, so what is this app supposed to do? What do you fear it will one day do? What would be the worst case scenario for you? And then of course I try to do that and I look through all the inputs and outputs and everywhere that there should be a security control and make sure it's doing the thing it's supposed to be doing. Sometimes if they let me, like, I'll go over their design with them and say, you know, have you thought about this? Or this seems like a bad idea or maybe if you did this, it'd be more secure and I offer like advice on it. And then I like to do just like a quick scan of their network, like, you know, their servers and the app and say, okay, so this is what I found really quickly. So please go fix all of this because I'm going to come back in a few weeks and then I'm going to be awful. So I've given you a head start.

Please get going.

But that's where you're in more of a waterfall model, I think, like every place I ended up doing testing it was a waterfall model. And so they're like, we're releasing in three months. Well, you better have me in next week then because.

The longer you wait to fix a security bug, the more it costs. So if I can help you in the design phase finding something wrong, it's so much cheaper than if it's after you've gone to production and something bad has happened. Or like a lot of times I would be hired right before, like right before they're supposed to go live. And like, this is awful because I'm that jerk that comes in, right? Like, the developers are like, she's just going to rain on our parade. And then I find things wrong, of course, because if no one's looked at security, there's probably going to be security problems. And then the developers have to stay up really late and it really, it kind of sucks for everyone. And then they're like, oh, don't worry, we're going to fix those in the next release. No, that's not what they're going to do.

So ethical hacker means I've been hired. Not by, like, could you go hack my ex boyfriend? Not like that, but like an organization is like, we want security testing done. So they hire someone that they can trust, will actually tell them all the things that they found wrong and help them fix it. And I usually tell people I'm a purple team person because I don't want to just give this report and then leave. It's really important to me to see things fixed and talk to the developers and sort out a strategy for them to. Sometimes they just have to go fix some things, but sometimes it's bigger than that. And I know a lot of penetration testers, which is like the professional name for ethical hacker, they get really sad because they'll go and they'll do a test and they'll give this big list of things that are wrong, and then they'll come back six months later and all those things are still broken and there's just more broken stuff and like, kind of disheartening as a job. That's what happens.

So that's actually a question. I get a lot from testers. They want to learn how to get into security and they always mention penetration testing. What skills do you think a person should have to make them a good penetration tester?

There's different types of penetration testers. There's network and infrastructure, and then there's web application, and then there's all the other things that exist when you do networks or infrastructure. If you hack a Windows 2012 server, that's R2, that as Service Pack 1, they're all the same. They're the exact same. And so there are tools like metasploit and other tools that. And things that you can memorize and things you can learn because it's going to be the same each time. So you do an NMAP and then you look at all the things that are there and then you just start an attack plan based on the things that you found. Right? But with web apps, there are certain things that it's likely you'll find, but each one of them is a snowflake because they're all custom. Imagine if you went to the hardware store and a 2x4 was sometimes 6 by 8.

And every single tool that you went to go use looks different. You have 20 different types of hammers and they all act differently and work differently. And then windows, some of them are circular and all of those things. When you create custom apps, they're truly custom. Certain things like you always look for. But there's lots of weird, special fun gifts that you'll find when you do penetration testing on anything that's custom. And those voter boxes are custom. They're not standardized yet. I believe that a lot of them are considering running Windows Embedded. I hope so, because it's actually. Then it wouldn't be custom and random and Windows embedded is like pretty darn secure. But anyway, that is another story. So if you want to learn about penetration testing, I feel like it would be a good idea to decide if you are more interested in software or if you are more interested in infrastructure, because there's kind of separate skills that you need for those for software. I feel that if you don't understand just the basics of coding, that it could be really hard. But I don't think you have to have been a software developer for 20 years either. Like, if you've taken a Java class and you just understand how basic programming works, I think you'd have like a huge leg up. I teach this workshop with my friend Nicole, often called how to hack your own apps. And I show everyone how to use OWASP zap, which is a penetration testing tool for web apps that's free and open source. And it is my belief that if all web application testers could just know how to run the automated scan on Zap and how to do it safely. So that's what our workshop does. We show you how to do it safely and not break things, and not break the law, not get in trouble, but actually get results. It's really not that hard to teach people to use it. I usually get them using it in under two hours, including explaining like safety and not burning down things. And.

And if every QA person could run Zap, that would just be so amazing because. And developers, if developers could just run the automated version of it, you can get so much more out of it as you learn more about security testing. And you can go further and further with it. But anything that you find with Zap is with the automated tools, really easy to find. And then that means that if someone is going to attack your site, they probably found those things. So those are the things you should definitely fix. So if you wanted to get started with learning how to be a penetration tester, I would suggest learning Zap because it's free and it's got a lot of things made, so it's easier to use for beginners. And there's a ton of documentation about it all over the Internet. And because people at OWASP really want people to use it and they'll probably help you if you ask for help. One last thing, if you want to learn how to become a penetration tester, I strongly suggest joining your local OWASP chapter and meeting people and then eventually trying to find a mentor. That's what I did, although my mentor found me and he kept convincing me to join InfoSec, which is not usually what happens, but having professional mentors is one of the best ways to excel in your career. And I have a bunch of them and I'm really lucky. And yeah, I definitely think finding someone that already knows this and who can advise you with specific questions is just. It's so valuable.

Tanya, we've mentioned a lot of resources here. You mentioned your show a few times. For folks that really want a list of resources or a place to go after they hear this podcast to actually dive in deeper. Do you have any resources you recommend?

I have two. One is a blog post that I wrote that is either all the resources that I use to learn about AppSec or ones that I've made to cover things that weren't there, and all of them are free. It's called getting started with AppSec and I'll give you a link for the notes. And then the other is I would love to encourage everyone to watch my streaming show or watch the videos after on YouTube. If you look me up on Mixer or Twitch, I'm shehackspurple and basically every Sunday I have guests on and we do a different application security or DevSecOps, which is just AppSec but in a pipeline type of activity. I'll talk about ideas of doing something when I do an interview like this or if I give a talk, but then on the show I'm like, this is where you click. This is the setting you want. This is why we chose this, and this is why this matters. And I can go like really in depth, in detail into each different little topic. And so if there's a thing that you want to learn, if we haven't covered it, send me a message because we want to cover it all.

Okay, Tanya, before we go, is there one piece of actionable advice you can give someone to help improve their software security efforts and let us know the best way to find or contact you?

I feel those are both the same and that's that you should read my blog. I'm. I'm writing up all of the things that I've learned in the past four, almost five years of working in security, trying really hard to attract everyone else into security. And I. I'm writing, you know, how to add security to your pipeline, how to add security to your system development lifecycle. You know, what is a secure coding guideline and like what should be in it. And then I'm publishing one really soon, as soon as I finish editing it. And I feel like I've tried to make it really, really helpful for beginners and at a level where you can ramp up really quickly. And that is a great way to reach me there or on Twitter hihackspurple.

Thank you Tanya for security Testing Awesomeness links to everything of value we covered in this episode. Head on over to testtalks.com 250 and while you're there, make sure to click on that Sign up for a free trial link under the exclusive Sponsor section to learn all about Sauce Labs awesome products and services. So that's it for this episode of Test Talks. I'm Joe and my mission is to help you succeed with creating Automation Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com for full show notes, amazing blog articles, and other automation awesomeness.