EP138 Terraform for Security Teams: How to Use IaC to Secure the Cloud

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website. cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button in your podcasting app of choice. You can follow the show, argue with us and the rest of the Cloud Security Podcast listeners all on our LinkedIn page. Anton, this is a fun one because not only is the video available on YouTube, I thought it was the perfect distillation of everything people wanted to know about Terraform but were afraid to ask.

Yes, but it's also a really good episode from the point of view of interaction of IT automation and security automation. Because maybe it's my ex-Gartner fear is that when I used to write about things like SOAR and other types of security automation approaches, I kind of realized there was this whole other world of IT automation, but neither myself nor many of the clients actually cared at all about this.

How could that be?

Well, that's the thing, because it's all very siloed at many companies. Security team wanted to automate, and they didn't even check what does Chef do, what does Puppet do, what does Terraform do, what does all this other stuff that IT built over the years that provides automation does. They said, we're going to buy a SOAR.

Oh, okay.

And it's interesting what results it leads to.

This sounds like another great example of one more tool syndrome versus sit down and talk to your neighbors syndrome.

Yes, I think so too. And I think that it's our tool. We are the security people, and these other tools are not ours, so we don't care about them.

Ah, yeah.

I think it's also a silo mentality of sorts.

Yeah. I wonder so much how I, as a vendor, can help people with that. But listeners, before Anton and I lament the common dysfunctions we see again and again in different themes, like fractal dysfunction, let's turn things over to today's highly functional And not Fractal guest. Everybody who's live with us today and everybody who's joining us later, welcome to the show. Thank you for joining us today. I'm delighted to introduce Rosemary Wong, Developer Advocate at HashiCorp. Rosemary, I'm delighted to have you here because this is a particularly, I think, cloudy episode on the Cloud Security Podcast, which, you know, sometimes we end up more security, sometimes more cloud. Today I think we will be a little more cloud. For our audience members who are new cloud. Could you give us the 2-minute picture on actually what is Terraform and how does it plug into the security lifecycle for cloud teams?

Sure.

So Terraform is an infrastructure as code tool, and it lets you build, change, and version infrastructure safely and efficiently. Infrastructure as code is overall a set of patterns and practices that borrow from software development space and allows you to apply it to your infrastructure. So some of the benefits that you get from doing infrastructure as code includes consolidating and standardizing all of the configuration of infrastructure. So from a security standpoint, it means that you can actually see what your infrastructure configuration is, rather than guessing as to whatever has been deployed. It's a little bit easier to audit and secure any infrastructure changes that go into that configuration. And by extension, it allows you to do static analysis of infrastructure configuration. So it means that you could possibly catch when someone's misconfigured a bucket or misconfigured a set of permissions somewhere. Before it makes it to your production environment. Overall though, I think the most important part of Terraform is that it's handling the provisioning as well as the configuration of cloud infrastructure. And it's not just cloud infrastructure, it can be any infrastructure API or API in general that you choose. So it doesn't have to be a cloud provider. It could also be a security tool. It could be a monitoring tool. It doesn't really matter. As long as there's a plugin for it, you're able to configure it.

So that's really cool.

So, let me ask you a strange type of question. So, this is really good, and I think Tim got it really, and I got it, and we sort of knew it already, but how do you explain it to somebody who is used to, like, pre-cloud way of doing things? Because we discovered that in some cases, security teams are kind of still in the— okay, I keep saying in the '90s, but really in the pre-cloud era. So, how would you explain Terraform to somebody who doesn't get the cloud-native stuff?

Yeah. So, I came from the network engineering space. So, we're gonna go with that.

Thank you. Okay. That's for network engineers.

Okay, so typically you would create a set of network commands and those would configure a network switch, for example. And so when you have all of these commands in place, well, you don't really know if the logic is correct or not. You copy and paste it and hope it's correct. And it does all of the changes in place on that network switch. What happens in the cloud, however, is that you have to be a little bit smarter about the order of operations in which things are configured. For example, if you wanted to create a server, You have to have a network first. And so what a lot of the logic behind Terraform is driving is what's the correct order of operations? How do we actually get a better sense of what's the current state of infrastructure and get to it? Rather than you as an operator or a systems administrator or security engineer trying to piecemeal and find the right logic to get to the end state that you want, you're allowing Terraform to help you get to the desired state, Which means more predictable set of changes, a lot more stability in how you're making infrastructure changes as a whole.

That makes a ton of sense. And plus the automation won't fat finger your config. So you take away a whole class of human-driven errors here.

Yes, exactly.

So this isn't just like the change from using shovels to backhoes. This is the change from shovels to GPS-guided millimeter precision backhoes.

Very close. That's actually a pretty good analogy.

Yeah. Your metaphors are so juicy today.

We had a whole conversation a couple weeks ago, listeners, about the difference between shovels and backhoes and gateways to hell. I don't think we're going to open gateways to hell here, right? That's not the goal?

That's not the goal, but you could do it by accident. No one's stopping you from doing it. You could Terraform apply and then open a gateway to hell. That's okay.

Okay. How would a security team or maybe our religious compliance team prevent somebody from opening gateways to hell? Are there ways to put anti-gateways to hell guardrails into Terraform? How does that work?

There are a couple of things within Terraform that can help you, but as a practice standpoint, you'll want to modularize your Terraform configuration, which means building in the best set of practices into some configuration that someone can self-serve and use. So a lot of the Terraform you'll see out there and you'll end up using is something that's going to be captured in a module and they'll have defaults. They'll have defaults for various purposes, for function, but also for security. And it allows you to give someone the opportunity to build something securely out of the box. And so that's one way that you can make sure that your Terraform is secure and you're not accidentally doing some kind of misconfiguration. The other way that you can approach it is through a series of either preconditions, postconditions, or testing, security testing.

Hmm.

This is where policy as code comes into play. But within Terraform, there's also these blocks called preconditions, postconditions. And what they'll do is they can test attributes before you plan to check some kind of configuration and after you apply a configuration as well. And so what that will do is it's a little mini test, right, that asserts whether or not this attribute is the way you expect it to be. And so some of these attributes may be, did you turn on encryption? Are you adding a security group rule that you shouldn't have added? Are these things the way that we expect it to be from a security standpoint? And then there's the policy as code side, which is a different kind of framework entirely, not within Terraform necessarily.

Okay, so hang on, hang on. Let me make sure I understood this. There's a couple of things, Before we get to policy as code, there's like teams can modularize this, they can reuse modules that trusted developers then push down to distributed teams. That's one option. 2, you can do checks at check-in time. And 3, you can build checks into the code itself.

Exactly. Yep.

Okay.

So that's like normal software engineering best practices kind of applied to the world of engineering your infrastructure. What's the policy as code now? Because that feels like we're bringing in a whole other can of worms.

Even before we go to policy as code, let me ask another tangentially related question. I just realized that some of the tech, like Terraform in this case, we sort of have 2 things, using it securely and using it sort of for the benefits of deploying, automating security. So, it's kind of like, how do you not cause problems with Terraform, but also how do you add security? It sounds like some of it can be used for security automation. I'm thinking, like, people build whole SOAR tools with UIs and stuff, but Terraform is already doing some of the automation at that level. So, Other benefits and other use cases of Terraform to automate security operation tasks, for example, or other security tasks? Yeah.

There are a number of them. I think the biggest use case that I hear with Terraform, and I use it every day, is actually around identity and access management. Let's say you wanted to grant someone additional access temporarily. Someone can open a pull request and say, I need this temporarily. It gets some review, it gets approved, merged, they get added. And then someone has the audit trail to go back and maybe take away that access if they need to. So you have the ability to either add users, add service accounts, add some kind of policy that you want additional permissions. And I think that's one of the more common use cases that we hear around Terraform. It makes it a lot easier for someone to decide they need that access and get the approvals that they need without adding additional friction. So that's one way that you can think about it. There are other reasons why folks use Terraform from a security standpoint. Some of them use it to set up security tools themselves, right? When you have different dashboards that you need to reproduce, if you have different configuration settings that you need for ingesting logs or putting them somewhere, that's also a reason to use Terraform as well. As long as there is a Terraform provider, which is effectively the plugin ecosystem, as long as there's a Terraform provider available to that tool, you are able to use Terraform to configure it.

So, it sounds like security teams that are trying to automate tasks using SOAR or using whatever other security-specific tech should really learn this. That's my impression, at least. At least, I do recall back from my Gartner days when this came up quite a few times. People came up and said, why are you talking to us about this SOAR? We can use Terraform. And it's like, yes, you can, but a lot of your peers are buying SOAR without even knowing that Terraform exists. And so, it sounds like there's a bit of a friction between security automation and, like, proper modern IT automation.

Yeah.

Is that pushing it too far or not?

No, but I think it's more of an indicator of the fear of automation, right? With Terraform, it's opinionated. You have to do things a certain way. It's really hard to say, I'm going to type out a domain-specific language and define my security configuration or infrastructure configuration. There's a bit of a fear cycle with automation in general. Automation can go wrong. How do I trust that I'm using the right provider? I'm using a module that works, or how do I trust that Terraform isn't going to apply something in the wrong order of operations and accidentally lock everybody out?

Hmm.

So, it's the fear of unknown. And I think that it is common across a lot of infrastructure. It's not just within security. I think one of the powerful things, usually when I work with folks who are interested in Terraform, but not sure if they're willing to adopt. Is to say, okay, well, how are you looking at your environments now? How are you getting an assessment of what people are doing in your infrastructure, how they're making changes? Are they secure? Do you know? And usually there's not a great end-to-end solution for them to do that. And so, there's a lot of incentive to move to something like Terraform so they can start bringing in infrastructure under management and make it a little bit easier to identify what's happening in their environments.

Okay, so I see so many upsides to this. I don't know, I have this mentality my life is terrible.

I don't know why.

I mean, the gateway to hell reference, right?

Yeah, right. And listeners, once again, we're recording a podcast the morning after a D&D session, so that's just like my brain gets a certain way. But what are some of the risks here? What are some of the downsides other than the fear? Like, how does this go wrong and how does it go wrong subtly?

Ah, yeah. So, Terraform has a lot of nuances, as in any tool in which you set it up, right? You have to be careful with Terraform because Terraform needs access to infrastructure or provider APIs, which means, do you give it administrative access? Do you give it editor access, owner access to all of these resources?

I guess you have to.

The thing is, you may not have to. And so, it's good to just consider some very specific role-based access control for Terraform. And so it doesn't need access to every API. Maybe it only needs access to certain infrastructure resources or whatever resources you want to do. So that's one thing in which it can go awry because you may decide to grant it editor or owner or administrator access, and now it has access to everything. The other thing is credentials and secrets management. Terraform has the ability to configure certain secrets somewhere, but it needs to know the difference of those secrets. So those secrets are in Terraform state. You should secure your Terraform state. You should also, in general, use dynamic credentials when you can. So the idea is that to avoid having those secrets in plaintext in Terraform state, use some kind of credentials injection, right? So early on, Terraform's able to retrieve it from a secrets manager. You can also retrieve it dynamically from an HTTP endpoint if you really desire, right? But the idea is that avoid putting those secrets in plaintext. Because someone can, if you don't secure your state, someone can go to state and extract that information. So be cautious about that. That's another place where it could go wrong. Dependency management of providers and modules. It's no mystery that someone could put a module or a provider out there that does something malicious. And if you're not careful, you could accidentally download it and compromise some of your infrastructure configuration.

So wait, there's supply chain? I know it's kind of an annoying subject to many because we talk about it maybe too much. But it sounds like there is a software supply chain, rather infrastructure as code supply chain challenges here. Oh, that's exciting.

Yes, it is.

That makes sense.

It is.

And so, of course that exists.

Yeah, it does exist because someone is creating this configuration. So you have to do your due diligence. Is this the right configuration? Is a provider doing what I expect it to do? We have Terraform Cloud, which offers private module registry, private provider registry, which allows you to have this ability to curate and store what you understand to be secure configurations, secure providers and modules, but you can create it yourself. You could put it in your own registry somewhere. The idea is that be careful where you're downloading a provider or you're getting a module from, right?

And so there's people out there doing like typosquatting on common patterns and all of that. How awful.

Yeah. You know, or like if someone just puts in an HTTP data source that's just going somewhere and you just don't know where it's going.

Love it.

Yeah, so it's possible. But the other last bit of this is that if you're not logging all the changes that you're doing in Terraform, right? So with Terraform, you have a list of changes that are being applied. If you're not logging that somewhere, you're not keeping track of it, well, you may or may not know what's happening.

Wait a second, is it not logged all of them by default? Because like, obviously I'm quite obsessed about logs, but in this case, it sounds like it should be part of the system, right? Is it not always on? I mean, we're not talking about how this other cloud charges for logs, right? Like, in this case, you don't charge for logging.

No. And so this is where it's helpful to have one, whether it's your CI framework, Terraform Cloud, something that runs Terraform. Don't run it locally, right? Because there are a lot of folks who do manage their infrastructure or manage configurations locally. And that's what we talk about logging some of these changes. There's logging in which you're pushing to version control, and by extension, you have a CI framework that runs it and controls how that change gets to production. But there are a number of folks who put it locally, and the result is that you don't know if someone has run it locally and has made a change. Unfortunately, Terraform, by nature, you can download and use right off the bat. But in order for you to scale it across teams, organizations, and to make it secure, You need to standardize the way it's applying and deploying those changes.

This also makes a lot of sense, right? Like, we developed SOAR tools so that we didn't have security engineers running Python on the server under their desk. So of course, the same thing happens here where we have to make sure that the Terraform apply isn't happening on the server under somebody's desk.

Precisely.

Yep.

Yep. Okay. And Anton, to say he's obsessed with logs is like an understatement.

Of the year.

Yeah.

Maybe not of the year, maybe of the decade. I don't know.

Yeah.

But it sounds like in this case, you Okay, I'm gonna rant for a brief second. Like, in this case, everything is just going your way. Like, it's a special system, it's modern, built securely by design, and it has logging. It's almost like if you don't log under circumstances, it's kind of your fault. Because I think that it's an automation that will make changes and you don't want the record. Like, why? What's wrong with you? I feel like it's easy to be judgmental about people who don't log under ideal conditions. It's not the same as logging in a system that was built in the '70s where logging was retrofitted in the '80s. Sure, I can see why you wouldn't want certain settings set because they would cut performance by 80%. I'm thinking of some database from years ago. But in this case, it's everything is going your way. So, if you don't log, it's kind of your fault. Okay, enough. Rant mode off.

It makes sense.

Rant off. You've got the next question.

Well, actually, you talked about some of the things that cause problems with security, but Is there like a crisp and concise list of security things to do? Like, I somewhat hesitate saying best practices because it's kind of a beaten term a little bit, but like, what are the top 3 things to do to have a secure experience here?

Yeah, I think the top 3 things, least privilege access, make sure Terraform doesn't have super, super, super access to everything. Just make sure to lock down how much Terraform is actually getting access to. redact and mask any sensitive outputs you have, right? So that could be passwords, et cetera. Just be careful of what you're putting in there that's sensitive and make sure you do your due diligence in redacting, masking, and marking them as sensitive. There's a way to do that in Terraform.

Yeah.

And so double-check that. And I think the third thing is standardized configuration through modules, because the more you set some opinionated defaults and secure attributes that you expect someone else to do, the more likely that someone else is educated in the correct security practice and they can do the right thing.

Use the modules, but know where they come from, right?

Yes.

Because you said use the modules. Before that, we discussed supply chain. So in this case, use the modules, great security idea, but know where they come from.

Yeah.

Okay.

Exactly.

Don't take it in a dark alley.

Know where they come from. No, don't just like randomly choose them from somewhere that you, you know, typed into a search engine.

Hey kid, got some modules for you.

It sounds like kind of a little bit like that, but, but people do use open source software like that sometimes, right?

Exactly. Yeah.

So I want to roll back up to what we were saying earlier before I cut you off and simplified things, then we got down a different track, and actually go back to something that was in the questions, which is policy as code. And so how do we intersect policy as code and Terraform and have the happy life of governance and controls adoption and all the good things?

Yeah, so Terraform in itself isn't necessarily going to help you secure your infrastructure, right? You have to have knowledge of how to secure your infrastructure. And I think there's a bit of a gap to allow an engineer who's configuring anything with Terraform to do the right thing. And so policy as code, in some ways, is able to help capture that knowledge and help give someone the ability to do the right thing by analyzing the static infrastructure configuration of Terraform. Now, it's a little bit of a misnomer with Terraform. It's not purely static, right? If we were to approach this from like a developer's definition of a unit test, right? You would ideally hope that policy as code analyzes the Terraform you wrote. So, the HashiCorp configuration language, you hope that it's able to get all of the attributes and information that you need.

Sounds right.

But that's not always the case. Yeah. You know, the unfortunate thing about, I think, infrastructure is that's not always the case. Sometimes you have different kinds of functions that's maybe subdividing CIDR blocks. You have different functions that you're doing dynamically, right? You're interpolating a variable from somewhere else. And that's not that easy to test from a policy standpoint. So Terraform plus policy as code means assessing the Terraform plan. So after Terraform has gone through the process of reaching out to the infrastructure API, making sure it gets the right variables, processing all of the logic that you've captured as someone who wrote the code in the first place, then policy as code can come analyze the static configuration of state. Not necessarily the configuration that you wrote, but the configuration of state. And that's where policy as code becomes a little bit more important. You have the ability now to say, okay, here's what my state looks like. I expect these things to be true. I expect these security rules to be in place. I expect this kind of network policy to be in place. I expect these users or these groups to be associated with these resources. Is this the case? And if it's not the case, then those tests fail. And now whoever wrote them or whoever's making those changes knows that, well, I didn't adhere to this policy. I need to go back and fix it. So there's where the cycle of policy as code becomes very important for Terraform. Terraform in itself doesn't necessarily know what's secure or not. So you need to tell Terraform or whoever's writing it, this is the secure way to do it.

And would a human do it or would a machine do it? I know it's a bit strange as a question, but if we are writing policy and then Terraform kind of complies with that policy to configure things securely and, and prevent prevent insecure configurations as well. What sort of skills the human who checks the policy needs to have? Because again, I'm kind of referring back to a few stories we had when security teams just aren't engineers and the organizations don't practice engineering-led security and their security teams aren't full of engineers. So, what sort of skill set would they need to have to read the policy and say, oh, this is okay, Or, no, this here would lead to Tim opening the gateway to hell if you run it. So what is the skill set needed?

Yeah, so it's someone who understands a little bit of the intersection of cloud configuration and security. And it's not always clear who knows that. Sometimes it's a security team, sometimes it's a cloud engineer or someone who just decided that this was really important and they needed to learn it, right? And so now I think it's becoming more and more important For that intersection to exist. But otherwise, there are libraries out there. There are policy libraries out there that are available that have benchmarks or various standards in place. And what those policy libraries have are already pre-written rules, right? Rules that assess Terraform for specific attributes. And they say, okay, this is the right way to configure it. So it could be saying, okay, here's a database. We must configure it this way in order for it to be secure. And those policy libraries are particularly helpful. There aren't too many of the open-source ones, but you can find them in different variants and different policy engine languages, like OPA is available out there with various folks, as well as some other testing frameworks as well.

So, this does make sense to me. And I think that it's with policies that are canned or prebuilt, the same challenge of supply chain immediately appears, right? Because, like, I have a policy that checks the security of a database. Where does that come from?

Yes.

So I think it's kind of funny enough, this episode specifically is increasing my respect for some of the supply chain issues.

Yeah.

Because I've been kind of, I shouldn't say dismissive, but I've been less excited about this than some other people. I'll put it this way. My excitement was kind of feathered.

But this also, I think, points to some bootstrapping problems.

Yes, it does point at bootstrapping problems.

How do you get this started on the right foot? And if you don't have the engineering talent, that seems really hard, or the engineering mindset. I guess this is maybe totally out of our question area, but what does a typical adoption journey look like? How does somebody get started and how do they get started right?

Yeah. So what they'll usually do is someone will look at Terraform open source. So they'll take it, they'll put it on their laptop and they'll say, oh, this actually serves the function that I'm looking for. It's helping me automate XYZ. And I'm able to understand it. Then they start diving into some of the documentation about how to put this into production, how to scale it. The next step that they're going to reach is, well, it has to be shared across multiple teams, right?

Mm-hmm.

Other teams have to be able to configure it. I want my teammate to be able to push infrastructure changes too. So then they add in the CI aspect, right? The controls that are in place in order for them to push Terraform securely and safely to production environment. Sometimes this includes policy libraries early on, right? They'll make the investment and say, here are a couple policies that we expect to be in place, and this controls quality to production. And then after that, after a CI framework, and after making sure that they're able to do this across multiple teams, from an organizational standpoint, people will start investing more heavily in modules. And that's where they'll start building some standards or templates around how they expect infrastructure to be configured and allow other people to self-serve and create that infrastructure with those modules in place.

That actually makes sense. And I think that given the time, and before Tim goes to a closing question, I wanna ask a relatively painful question. And it's a painful question about teams that sort of still stick to the lift and shift mentality, kind of pre-cloud thinking in many regards. And I know that Terraform sort of exists in a different world of a modern cloud infrastructure as code. But what do you, as an advocate for this, typically tell the teams who kind of want to do cloud security the old way and not the cloud-native, cloud, cloud, sort of a modern cloud way? Like, how do you talk to them? What do you tell them? Get out of a cave and all that. Like, what are the things you say?

And what I love about this question is we ask so many guests this question from so many different perspectives. So, this is like, you're one of the most cloud-native people get this question. So, how do you possibly cross that chasm?

I don't know. That's a new title here. No one's ever told me that I was one of the more cloud-native people. But one of the things I say is that you take small steps, and it doesn't make your life better, and doesn't make your life easier. With cloud-native, or when you go to cloud infrastructure in general, it's really easy to get infrastructure and any resource. And so, it makes it very difficult for you as a security engineer or someone who's managing something, I guess, in the old way, to control or to provide guardrails around how someone is going to use cloud. I think someone best told it to me like, the cloud is like the wild, wild west. Every developer gets to do whatever they want. And it's not making our lives easier because we can't audit what's going on. We don't actually know what they're doing, what they're using. Cost is a factor. And we spend more time sitting here reverse engineering what someone's done than us actually putting the right guardrails in place. And so, yes, it's painful. It takes time. It's okay to be scared of it. But what short-term investment you make into shifting towards something like Terraform and thinking about infrastructure as code will help long-term in that you won't have to experience the pain and reverse engineering of what someone's done. And ultimately, it provides stability and security to the overall system.

That makes a lot of sense. So with that, I have to ask our traditional closing questions. One, do you have a tip to improve people's security journey using Terraform? And 2, do you have recommended reading? And that could be anything. We've gotten everything from Mary Oliver to Italian quantum gravity physicist books.

Oh. And Dune, of course. Dune. They got Dune.

Okay.

And Anton's blog.

Okay. So it could be any book. Well, I guess for folks who are looking use Terraform to automate security, I would say iterate. Don't expect to automate everything perfectly from the very start. Choose something low impact, practice the patterns. Do that first, get a sense of how it works. What are the gaps? How does it fit for you? Especially in security, right? Because you're going to have a lot of different things to think about. So, take something really small and start there first. Don't get ambitious and shift everything to Terraform immediately.

It's a great answer.

Yeah. As for a book, There's a book I think is called Mindfulness of a Sloth.

Wow.

It's a very cute illustrated book that's about mindfulness. Oh, A Sloth's Guide to Mindfulness. There we go. A Sloth's Guide to Mindfulness. It's a very cute illustrated book about a sloth teaching you mindfulness. And I think we could all use a little bit of that these days.

That's fantastic. Well, Rosemary, thank you so much for joining us today. This has been delightful.

Thank you for having me.

And now we are at time. Thank you very much for listening and of course for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter, @anton_chuvakin and @_timpeacock. Tweet at us, email us, argue with us, and if you like or hate what you hear, We can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Thank you, Roni.