Securing DevOps: Security in the Cloud with Julien Vehent

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Security Podcast. We're all getting together to learn more about security testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Security Podcast. Today we'll be talking with Julian Viant, all about security in DevOps. Julian is the author of the book Securing DevOps Security in the Cloud and is a security architect at Mozilla. With over 15 years of experience, he specializes in web application security, cloud infrastructure, cryptography, risk management and more. You don't want to miss this episode. Check it out.

It's no secret that the number and complexity of applications are growing. But what is alarming is that 90% of security incidents result from exploits against against defects in software. The need for better application security has never been higher and Micro Focus Fortify is here to help. Fortify is the recognized market leader in application security and it is the most comprehensive and scalable application security solution that works with your current development tools and processes. With Fortify, you could start security applications in a single day, including custom code, open source or commercial components and scale as your needs grow with an on premise as a service or hybrid implementation. Check them out. Head on over to micro focus.com app security. Hey Julianne, welcome to the Guild.

Hey Joe, thanks a lot for having me.

Great to have you on the show. As I mentioned, we're really going to dive into your book, but before we get into it, could you just tell us a little bit more about yourself?

Sure. I run the Firefox Operations team at Mozilla Firefox Operations Security team at Mozilla and we focus on the security of the back end infrastructure of Firefox. Firefox is a browser, but it's also a lot of cloud services. And every time you use Firefox you end up talking to a fairly large set of those services hosted in various cloud environments. And the role of my team and my role is to keep those environments as secure as possible. So I spend a lot of my time in the cloud. I spend a lot of my time thinking about threat models and attack vectors and secure software and application security and infrastructure security and all that fun stuff. I come from more of a systems engineering and network engineering background years ago, before we had the clouds and slowly made my way into security and to the point where it devoured entirely my occupation and, and that's what I do day to day now. And I'm also the author of a book called Securing DevOps which is really a compilation of techniques on how to run secure cloud services, particularly in aws, but that can also be applied to many other cloud environments. And that explains a lot of the techniques that we developed and implemented during my time at Mozilla.

Now, writing a book takes a lot of time. I ask authors this all the time. Why did you feel the need to write this book?

I think it's some sort of a bucket list item. You have to do it, right? If you're that kind of person, you have to do it and you're going to do it at some point. And this was the right time for me to do it. I had a topic I was interested in. I was pushed into it a little bit by my family on one side and my editor reached out on the other side and I figured, why not? If you're going to do it, might as well go all in and write a book that is interesting and comprehensive. So that's what I went for. But it did consume all of my free time for almost two years. So that's not necessarily something I recommend people do.

No doubt. Well, it shows. So I'm really, really glad you did write it.

Thank you.

So the book is broken out into three main sections. You have securing DevOps, watching for anomalies, protecting services against attacks, and maturing DevOps security. So I thought we'd go into each section and maybe pull out a topic out of each area. The first one, I guess, is why do you think security must be part of a really important part of your product development?

Well, I think it's the only way we're ever going to have secure product and secure services. I think trying to. The same way when you build a house, you don't call your security engineers at the end to add reliability and safety to your house. It's part of the design. That's how things are built to be safe. It's the same with software. You have from the very beginning to consider how it can be misused, what impact it could have if it gets attacked, etc. Etc. And bake those controls directly into the software, the service, the infrastructure themselves. And, and so gradually over time, what we're going to see is that the organization that resists attacks and that provide secure software are the ones who are taking security seriously and integrate security considerations as part of their software design. We use the term shifting left in the industry, which really refers to moving the security discussion to the beginning of the software engineering life cycle, the sdlc. And that's really what it's about. If you want to have secure Software. If you want to have secure cloud services, you need to take these considerations into account from the very, very beginning.

You work for a large organization. Just curious to know how do you get buy in to actually do this approach? Because obviously it's not like free.

It isn't. What we find is that it gets cheaper over time. Because at first what most organizations that adopt this type of security integration do is that they first focus on the technical aspects. So they focus on automated testing, they focus on integrating security testing in CI, cd, et cetera. But over time they start creating a culture of security in their organizations. They start talking to their developers earlier and earlier. They start talking to their operations team earlier and earlier. And the, the cost of having those security discussions and of shaping software to be secure reduces over time. What we found doing this at Mozilla is that it is actually cheaper to have security discussions early on now than it is to have them further down the pipeline or to have them like we did a few years ago. So it ends up being a cost savings.

Really process for most organizations that do it right. And once the rest of the organization, the developers, the product managers, the operators realize that, they just buy into it.

Interesting. So if you worked in an agile team and someone starts a new feature as part of their sprint, is security part of the definition of done or is it like just understood that we need a plan for security for this particular feature before we release it?

I think it's part of the definition of done.

We started using security checklists specifically a few years ago. And what we realized over time is that we were giving the security checklists to the engineers and the project managers would actually take those checklists and add them to the acceptance criteria for the projects without us asking. So they really considered the project to be done to be feature complete or the sprint to be feature complete once security was also done. And that's really, it's not even something we asked for, it just came from the culture. Right. And that's a better model. I think that's where we started seeing the best results.

That's awesome. So it sounds like once you have buy in, you have this culture that's embracing it. What are some steps you think folks need to do to create a bare bones DevOps pipeline for the security efforts?

That goes back to why the book is structured this way. I think the first phase, and I tried to model the book around the idea of a three year plan. So imagine you're starting, you're the first security engineer in a fairly small organization and this is year one and nobody else is focused on security. What are you really going to work on? And chances are you're going to first look at the infrastructure security and you're going to look at the technical aspects of running services and websites in that infrastructure. And that's fine because if you can secure the infrastructure, everything else will become automatically a lot more secure. So the first year is really focused on.

Infrastructure and after that application security as well, but at very technical levels, automated testing, how do you run your test in CI cd? How do you run your test continuously, how you do, do you communicate the results of those tests to your developers, etc, etc. And the second year is when hopefully for some organization it goes faster, for some it takes longer. You can start looking into fraud and abuse and detecting malicious activity inside of your environment.

Third year is hopefully you have mastered fraud detection and incident response and you have good infrastructure and application security. And you can start thinking about more strategic items like risk management or threat modeling. I wouldn't recommend that someone who gets started on this will start right away from a threat modeling point of view. Mostly because threat modeling is only really effective once you have a security culture in an organization. Otherwise you just, you're just that security guy in a corner of the room telling people they're going to get hacked by the NSA and no one cares. So you need to have created some buy in before you can effectively do that threat modeling.

Are there any tools that you use or you recommend people use in their DevOps pipelines to help them with their testing?

So the tool we've probably invested in the most is the OWASP Z attack proxy Zap. And we use that extensively throughout our entire set of services. To audit, we have something we call the ZAP baseline scan that essentially goes look at our applications for default security headers, default security parameters that we want to have everywhere. That's a tool that Zap is very popular. It has a lot of integrations with various CI CD platforms. I know a lot of people use it with Jenkins for example, and most people are very successful with it. We have grown. There are a lot of tools that do infrastructure security. I like Scott Piper's cloud mapper for aws. I think he does a great job growing that platform and it really helps find issues in AWS environments and you can automate that to run continuous tests against an AWS account. So if you start putting these tools together, there isn't one tool that does everything. And I don't think there should be. I think it's Better to have specialized tools because you can compose them to match what your organization is using and then automate these tools to, for example, automatically raise issues when they find a problem or block a production deployment. So that's what we do. And then we've grown a number of tools internally. We implement and we built our own unit testing framework for infrastructure tests, both AWS and Google Cloud. And we added Heroku, I think, and we have Azure as well. We can run things like check that all of the database backups are encrypted, for example, and we can run those tests continuously. It's harder for me to recommend one specific tool that does this well, because what I found is that it is often better for organizations to build their own tools and build their own test suites and to actually collaborate with the operations teams and the developers on what those security tests should be. Because coming up with what we used to do in the past is take a scanner and it would output a report with like 200 failures and then go to your devs and say, can we fix those?

And they wouldn't they look at it be like, what? What do you expect me to do with this? So rather than doing it this way, I prefer the approach of cherry picking a few areas where we want to improve. Working with the devs and the ops, getting buy in, then writing the tests, then getting everything in compliance with the test, and then moving on to another level of test and using homegrown tools is better for that process.

You did mention checklist and people always ask me for checklists. Is that part of the book? I didn't see your security checklist in.

The book, so I probably didn't mention it in there. The way we use security checklists is that we essentially have a master checklist that we will give to developers every time they start a new project. And that has everything from like infrastructure, security.

Security headers, TLS configurations. Common issues we find when developers implement APIs in AWS, for example, like if you proxy web requests, you have to protect your instance metadata endpoint. And we found that issue in enough applications that we put that in the checklist, like, hey, watch out for this, it's a common problem. And so that checklist is a mix of industry best practices, but also issues we've learned about both from our own security work, but also from our bug bounty programs. And we want to tell the devs to watch out for them.

That master checklist usually gets split up depending on what the developers are working on. So we will, when they start a new project we'll take the checklist and we will discard the stuff that doesn't apply to them. What we commonly discard is the entire infrastructure security section, because most developers will host their applications in the core infrastructure and so they follow the paved road model of inheriting all of the controls that exist in that infrastructure. The default TS configuration, the default log management, the default secrets deployment, et cetera, et cetera. And so they don't need to worry at all about this. Over time, the checklist that we end up giving to developers actually becomes shorter and shorter because we're implementing better default and better core infrastructure. Better paved road for everyone. And so they have to worry about less and less of the security perimeter. The people who want to do their own thing and host their applications elsewhere, they have to worry about the entire thing. And that's actually a good argument to discourage them from doing shadow. It's like if you do it yourself, you have to do all of this yourself, which is a lot of work. If you host it properly, you don't.

So once you have your developers, they have the checklist, they're trying to write more secure code before it actually gets to. When you know someone's just hacking it when it's in production. What are some things you could do to know if you are being hacked? Like are there any ways you can look at log files, look for anomalies and. Or to protect yourself from those kind of attacks or intrusions?

Yes. So that's. This is an area where security teams spend a lot of their time. In most organizations that reach a certain size, you will find security teams that are focused on fraud detection, on incident response, etc. And threat hunting, all that stuff. What I would recommend developers do from the get go is standardize your log formats. First of all, that's the most important thing. Nowadays most applications are built inside of Docker containers. You can output application logs just on the standard out of the Docker container. Just use a standard simple, maybe a JSON format for application logs and take those application logs, take the web logs and put them in a standard log management facility. If you're in AWS, you have CloudWatch logs, if you're in GCP, you have StackDriver, et cetera, et cetera. Every single platform has a way for applications to publish their logs into a central place. The very first step to take is just to have some very basic rules around.

Traffic that doesn't look like the normal pattern. So if you know that, for example, a client should never make more than five requests a minute, then just have an alert on that and you'll find a whole lot of stuff that isn't normal traffic that may lead to detecting a bunch of IPs that are malicious. And then from that point on, the fraud detection logic becomes more and more complex over time, depending on what the application does, depending on what the traffic, whether it's a public or internal application, et cetera. We use very, very different techniques, even for applications that may look the same from the outside, but in fact end up doing very different things. At that point, though, if you're running a service that is popular enough that you're getting a significant amount of fraudulent traffic.

You probably should have someone expend a significant portion of their time fighting fraud on this. But it's always easier to do if you have standardized logs and they're going to a standard log pipeline and it's easy to query and visualize and run some simple detection scripts on those logs.

Great. So it sounds like it all depends on certain things. Is this where risk management comes in? Like where you should focus your energy? Because I assume it's a lot of things you could do with security and you can't do all of them. So is risk management something that comes into play of maybe we should put extra efforts into our logs or detecting intrusions?

I think so. Varying degrees of complexity. When you say risk management, people usually think of those massive spreadsheet that NASA would do to, for example, to figure out what could go wrong with sending a rocket to the moon or something like this. And it doesn't need to be that complicated. In fact, most people, particularly product managers and business managers, are just naturally good at risk management. They do it in every business decision that they make, essentially. And so applying that to security isn't very different. So it's fairly simple for a team of engineers to just sit around the table once in a while and look at an application and ask, how would we attack this from the outside? What are the risks? What's the most dangerous thing that could happen to this application? And that usually gives a lot of interesting pointers. You may, for example, have one application where the most dangerous vulnerability would be risk would be to leak the data. So you have a confidentiality risk. And so what are you going to build? You're not going to build a rate limiting.

Security control. If your risk is leaking data, you'll probably go look at your database permission, you probably go look at SSH connections to the production infrastructure, all of that stuff that may lead to leaking data or accessing data fraudulently. Another application that may be very similar in design may in fact have mostly an availability risk that if it's down then the rest of the company cannot function. And so you're losing money very, very quickly. Maybe you don't have a confidentiality risk there, but you really, really want to be fully available at all times. And those discussions are not complex in nature. There is no risk framework involved with just asking what do we need to be worried about here? And the solutions usually come naturally. Then there's a question of how do you prioritize this? And that's where. Well, if you have a really, really long list of non obvious priorities, it can be a bit complicated and you can get into more.

Complex risk management frameworks to help with that. But I would say for most organizations you can always start with what are the top three top five problems that we want to protect against and focus on those. And for many organizations that will be, well, a break in is a major issue both from a reputation perspective and maybe from a regulation perspective. And so we need to focus on protecting all of the access points.

That's the kind of exercise that we like to have internally with our developers. Fairly early on is asking those questions. But honestly, I think any organization should be having these discussions regularly.

Absolutely. So you used a term in the book that I haven't seen before. I've heard of test driven development, behavior driven development, domain driven development. But you have something called test driven security. Is that something you coined? And if so, can you give an example of what test driven security is?

I don't think I coined it. I think I've heard it elsewhere before. Perhaps the book has the more fleshed out definition of it. I think the main idea behind it was really to push security engineers to define the security controls that they want to see implemented everywhere from the beginning, and then to let developers essentially go through the failing tests and fix them over time. So the same way you would do tests driven development, you write your unit test first and then you go implement the functions. Here you would have your security test, and if you're starting a new project, it will have none of the security requirements implemented. So it will fail all of the security test. But the goal here is to tell the developers which tests are failing and to have enough documentation that they understand why the test is failing and how to fix it and why that's important so that they can go fix it. And by the time the application reaches production, it has all of the passing tests done. I'll be honest, I've had mixed success with this approach. What we found is that the technical implementation of test driven security with security tests running in CI CD continuously hasn't really caught on. And I think it's a mix of like our security tools don't really work perfectly in those environments, are still kind of heavy. And also there's always this weird part of the code isn't the security code isn't necessarily owned by the devs and so it doesn't get full adoption. What works well is, like I said, the checklist works well and letting developers essentially self assess when they feel they're ready. And we do that by exposing our security tools to them. For example, observatory.mozilla.org is a website where you can request a scan of your application or website and it will tell you if you don't have a good content. Security policy will tell you if you're missing security headers. We found that developers like to use sites and products like these and that seems to be more successful. But the core concept of test driven security, where you essentially will fail your CI if you don't have all of the tests implemented, that's still something that we haven't really found the way to make it click everywhere.

Great. So another technique that you mentioned, you actually mentioned that you took the feedback and incorporated into your checklist was bug bounties. I've heard of bug bounties before, but how does it work with security?

Well, it's pretty easy. You basically tell people to come hack you for money.

It's basically how it is.

Mozilla has had a bug bounty program both for Firefox itself and for the web services we run for maybe 15 years now, maybe more than that, I'm not quite sure, but it's been a long time. So it's always been part of the culture internally and we really have a good relationship with a lot of researchers. What we got from it is essentially when we create, even on our existing services or new services, when we create new application new services, we can, without putting any money up front, expose those services to a population of fairly skilled security researchers and they will go look for bugs. And if they find bugs, they will report them to us privately so we have time to patch them and we'll pay them money to thank them. So the amount that we pay depends on what they find. So we have a on if you search for Mozilla Web bug bounty, you will find a page that describes the payouts per categories of vulnerabilities. And we pay more for our core sites than we pay for our non core sites and we pay more for remote code execution than we pay for like a cross site scripting attack. So it varies depending on the vulnerability found a lot of organizations so we run our bug bounty program ourselves, but a lot of organizations rely on third party vendors to do that. There are many of them, Bugcrowd and HackerOne are two of them that are fairly popular that will enlist your organization and expose it to researchers and help triage the large number of vulnerabilities that get found. There is. I will say though that before starting a bug bounty program, it's always good to check in with the lawyers because there are some regulations that need to be taken into account. We have published some of our approaches on the Mozilla Security blog of doing what we call safe harbor, which is basically a safe, legally safe environment for researchers to come essentially research vulnerabilities against our website without risking.

Essentially legal action from us. So there are rules to be followed on both sides, but it's something that organizations should consider before starting a built guarantee program.

Okay Julian, before we go, is there one piece of actionable advice you can give someone to help them with their security DevOps testing efforts? And what's the best way to find contact you or get our hands on your awesome book?

Sure, the book. So that's a pretty easy question. You can go to securing-DevOps.com and it will automatically send you to the book page and you can get your paper, copy, ebook, copy or both from there directly. The piece of advice I would give to people implementing a security strategy in their DevOps organization is to break the barriers between teams as early as possible. The most important aspect of building a successful security organization inside DevOps team is to be able to talk to people often and about everything. And when I started in the cloud services group at Mozilla, I was actually a member of the operations team reporting to the operations manager. I was in the room with all of the systems engineers and operations engineers and we were working on incidents together and we were working on operations together. There were no barriers. We built trust that way. And one thing that often lacks in security organizations is that they're not necessarily trusted by their peers because they're too far away, because they're trying to do everything through tests and through bug reports and et cetera. And that's not how you build trust. So if you want to really build a successful security organization, break the barriers, go work directly with the developers, directly with your players, share their successes and their failures, and really create a cohesive group this way.

Thank you Julian for your security testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com S10. And while you're there, make sure to click on the Try It Today link under the Exclusive Sponsor section to learn all about Micro Focus Fortify, which is the recognized market leader in application security. So that's it for this episode of the Test Guild Security Testing Podcast. I'm Joe and my mission is to help you succeed in creating end to end Full Stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Security Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.