EP56 Rebuilding vs Forklifting and How to Secure a Data Warehouse in the Cloud

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the product manager for threat detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and esteemed member of the cloud security team here at Google. You can find and subscribe to this podcast wherever podcasts are available, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you like our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show and argue with us on Twitter as well, twitter.com/cloudsecpodcast. Anton, today's episode is about data warehouses, which I found to be way more exciting to talk about than I thought it was going to be.

That's exactly right. And I think that the fun part is that we are talking about people who are piling up huge, huge volumes of data into Google Cloud, and then it kind of gives birth to special security requirements. So this sounds like a very fascinating setup for an episode, right? You're not just copying a little bit of data. You're copying possibly massive volumes of data. And then you have to account for regulated datasets, for threats, for controls. It's way more fascinating.

Yeah, we said some of the words we tend to avoid on this show too. We used the P word. We used the G word. It was a good episode, I think.

Oh, you're right. We did use the P word. And of course, I got immediately skittish.

You did.

Because I thought that maybe 10,000 lawyers would show up out of the woodwork and would tear us But we did say the P word. Yeah.

And so far, so good on the lawyer front. So with that, let's welcome today's guest. We are joined by Erlendur Loður, or ELO, not to be confused with the chess ranking system, who is a security and compliance specialist here at Google Cloud. We're talking about securing data warehouses, which I think is a fascinating problem because data warehouses are one of the, in my mind, sort of canonically obvious use cases for cloud technology as opposed to doing it yourself on-prem. You wouldn't believe the stories I've heard of users having to provision, you know, yards and half miles of data center space for this problem. And instead, it's a couple of API calls into gcloud to get it going. From a starting point, Erlendur, what are the things to think about when it comes to securing a data warehouse in the cloud? Where do you even start with this problem?

Yeah, absolutely. And thanks for having me here on the show today. Super excited to be here. So really, you might think about this at first blush about, hey, maybe just take BigQuery and use the built-in controls that we have with BigQuery and call it a day. But really for me, it's more than just protecting the data within BigQuery itself. It's about that whole ecosystem, especially when we think about things from securing data at scale. You know, in the data analytics world, they often think about the 3 Vs of data in a sense, like the volume of the data, its velocity, and its variety. And you have to think through as well how that data is moving into the cloud environment. It's usually typically done with an ETL type of process. So what's doing that transformation and how reliable that is? And so when I think about the security of a data warehouse at scale, I think about 3 different areas, really. I think about, first of all, it's sort of the business objectives that we're wanting to achieve and how security can help with those things. So I had a healthcare customer who has some PHI data that they want to bring into cloud. And so with that particular data, they needed to understand how to de-identify that because they had a requirement for the data in the cloud to be de-identified. What did that data look like? We kind of worked through those different things, and we understood that it needed to be in FHIR format. And there were certain fields inside of that data that had unstructured doctor's notes inside of that, that they needed to be able to make sure that was de-identified as well. Really being grounded and understanding what how that data is gonna be used really helps with the initial part of securing the data warehouse, understanding the business objectives. The second thing I think about then is the data governance around how this data is going to land in the data warehouse. And so what does that compliance landscape look like, if you will? Typically this data, it may have your confidential data inside of that. It might have PI information, but more so than that, it might have the notion of your crown jewels, if you will, where—

Mm-hmm.

this data might have that competitive advantage that you need in there. And so it's really understanding how do you protect that information? What's the governance around that within this cloud environment now? And then lastly, what I'd like to think about too is using sort of like a risk-based approach, if you will, one where you think about the blast radius, you think about MITRE ATT&CKs and the TTPs and what might happen, how do you layer in security around that? And one systematic way to think about that too might be through Like using a threat model, if you will, and to really help you get a sense of what are those risks and what's the threshold that you have for those risks.

That's interesting to me because, and Anton's probably going to laugh at my naivete here, but you put threat modeling last and preceding it were governance and what sounded a lot like a compliance requirement. And so in my day, I spent all day thinking about threats, but it's interesting to put that last.

Yeah. And then frankly, I wanted to pick on the same theme exactly, that a lot of this is kind of about regulatory requirements and what the auditors may want and what the customers may think they need to do to satisfy somebody. So, and of course, you mentioned PHI, which is valuable to attackers, but in most contexts, I've seen people bring up compliance when they discuss it. So, to switch gears a little bit, I know it's all important, but what are the expected threats? That is threats in addition to auditors.

Those are the biggest threat often.

Other than auditors. So, tell us about the threat models and what are the expected threats to a big data warehouse in the cloud if you exclude the auditors? For a second.

And for our guests who don't think about healthcare data all day, what's PHI?

PHI is personal healthcare information. Some of those things are defined by Safe Harbor that you can go through, look up. I think there's like 18 of those per se, but those are some of the types of things that we consider as PHI. Now, that's a great question too. It's very interesting to think about that. When I talk with different customers who've asked me about securing their data warehouses, usually one of 4 or a combination of 4 different threats comes up. One of them is elevation of privileges. And really, that's about trying to understand if an attacker could get ahold of some sort of identity that can do more than what they're able to do initially.

Sorry, attacker here means an insider or necessarily an outsider or either?

It could actually be both from that perspective. Maybe it's a malicious insider, but typically it's someone that maybe has misconfigured something. But it could also be a human, or it could be like a machine identity that you want to think about there as well. We spoke a little bit about auditors, for instance. You probably don't want an auditor to be able to see PHI information about certain things, or maybe even your SRE team to be able to see some of those things within your BigQuery datasets. Another expected threat that I typically run into or have conversations with customers is data exfiltration. Maybe that's because of, like we were just talking about—

Which is data theft, right?

Yeah.

Because when you say exfil, that means somebody's doing it. It's not just something that's permission is opened by mistake. Data exfil implies somebody's doing it, correct?

That's right, where they're probably trying to get a hold of that sensitive information and send it somewhere else, or think about how to capture and keep that information. It's a great point.

You said something in there that I'm sort of surprised is even possible, and I wonder if you could expand on it for the audience a little bit. You said, I hope I understood this right, keep your SREs from seeing the data. How can you keep people who have root in prod from seeing data? That sounds like magic to me.

Very interesting to think about it from that perspective. When you have these SREs from root and prod, when you're thinking about from an on-prem sort of environment infrastructure, if you will, a lot of the services that we have within our data warehouse are more about these sort of serverless technologies, if you will. These folks hopefully will have a different set, at least we've defined a different set of identity and access controls that we've put in place so that we understand and we make sure that those folks that are able to, you know, we separate the ability to read and look at data versus the ability to sort of do the administration and management. piece of things. So I'm hoping that helps give you a little bit of insight from that perspective. But a lot of these services that we're talking about don't have something that you can traditionally like log into and try to gain root access to things.

So to me, that's kind of the magic of operating in cloud, right? Where we have a permissions model that doesn't have this concept of like root on the database box. It's just, you can divide your privileges differently enough that there's threats on-prem that really aren't as relevant in cloud. by the nature of the surface. That, to me, is close to magic.

That's exciting. Another threat that I often talk to customers about, too, is then information disclosure. That's really about having folks within your team be able to see the data that they're allowed to see. So we talked a little bit about auditors. What are they able to see? They probably, like we were talking about earlier, should not be able to see that sort of information.

So this is not theft. In this case, just to balance quickly— and sorry for interrupting— data exfiltration means somebody is taking the data. This one is kind of more about there's no theft, there's no mistake in permissions, but somebody is seeing the data who shouldn't do it.

That's correct.

So it sounds very close to the P word, which I promise to never pronounce on this podcast. So is this a security issue or is it other than a security issue?

Yes, that's probably other than a security issue, if you will, from that perspective. So it's a great observation there. It's still often a topic that'll come up in some of these discussions that I run into with customers. Additionally, one last thing of the 4 items is usually the availability of your data. A lot of the regulatory frameworks will have a notion of business continuity and availability, or things like disaster recovery. But if your data scientists are trying to create and surface some real-time information from your data warehouse, you really need to be able to understand how that data is going to be available and how quickly it'll process and flow through your data warehouse environment. Hopefully, That gives you a sense of some of the threats that I often discuss with customers.

That absolutely makes a ton of sense. We like to have practical advice on this show as much as we like to get into talking about Karl Popper. What's your advice for somebody who's planning their data warehouse project in cloud? You know, how do they get started with this? What should they do in their planning to make sure things go as successfully as possible?

I like to think about it in a couple of different ways. One of the first things I think about is where's the environment of the data warehouse? Where is it going to be deployed? Is it being deployed in a place in your cloud environment where you've already carved out and had the ability to deploy maybe your security foundation blueprint, for instance, where it's already established the necessary guardrails from org policies to IAM and network controls. And then you're just deploying the data warehouse blueprint on top of those sorts of things when you're thinking about how you can use that underlying infrastructure. Or maybe you've already spent a lot of time hardening your infrastructure. So you want to be able to understand how your data warehouse will work within that hardened infrastructure that you may have. Or maybe one of the things you're thinking about is deploying a data warehouse sort of in a POC type of standalone environment where you want to evaluate a lot of the controls out of the box for a particular blueprint. So really understanding where that data warehouse is going to land in your environment is gonna be an important aspect. The second thing to think about too then are the people that are involved here. So for instance, if you think maybe your data engineering team is gonna be responsible for deploying this and your data engineers may not have as deep technical insight and understanding of security as your actual security team from that perspective. You might wanna understand that and also understand where your security team is. Did they just finish up maybe handling an incident response, or maybe they're just finishing an audit process like what we were just talking about? So really understanding the people that are gonna be involved and available to help you deploy your security, your data warehouse, it's going to be an important aspect. The third thing I think about too is then sort of the business timing of all of this. When do you need that solution? And maybe we could kind of take a play out of our friends from AI/ML where they We've really thought through the journey of a data scientist for supervised learning, if you will. If you take a look at some of the neat things that we've done within Google, we've got these pre-trained models where, like Vision API, where really you need something fast, you don't have any data to do some of the training, but you have— you need to make use of some of those pre-trained models. And then on the other side of that, you might be able to have your data scientists that have deep expert knowledge and understanding, and they can use Vertex AI, understand MLOps and TensorFlow. and they can build their custom models with this rich data, training data that they have. Or maybe they need something in between where they've got AutoML or BigQuery ML to do some customization with sufficient data. Likewise, when you're thinking about from a data warehouse perspective and looking at the data that you have, if you're able to take something like a data warehouse blueprint, if you will, and think through all of the different items that you may need, if you don't have the time to build out an entire secure environment, you might be able to take a data warehouse blueprint as is and deploy that. If you need to customize things, maybe you don't want to use as many service perimeters, maybe you're not ready for streaming data into your environment, or you've got your own Dataflow Flex templates that you want to use, maybe use those instead of the ones that we've kind of thought through. Or if you've got that deep security knowledge and your architecture team has an understanding of how to not just use the individual security capabilities, but layer them together across the entire ecosystem and environment, maybe they can then use something like the Data Warehouse Blueprint as an architectural checklist. to understand some of the best practices that we've developed and kind of comparing that to the decisions that they have made. And then the last thing I think about too are then the future plans that your data warehouse has. If you think about using a blueprint for building a house, for instance, you may have the notion of an unfinished basement that is already kind of built into the environment because it's harder to obviously add in a basement after you've built a house and those sorts of things. One of the other things to think about too, when you think about the future is, are there appropriate hooks in place so that way that you can handle any changes that the business may need. So for instance, I was working with a customer in the financial services industry where we were talking through re-identification and would there be any sort of use cases and needs for that. Immediately they didn't have that particular need, but they were really interested in some of those capabilities that we had. And they said, well, that'd be kind of interesting maybe for 1 year or 2 years down the road. But for right now, what they needed to do is just simply de-identify that data. keep it de-identified. So hopefully that gives our listeners a couple of things to think about when building out a data warehouse.

Elo, that was an incredible answer. If I were to summarize, first thing you want to do is make sure you've prepared your cloud terrain and got everything ready to go. You've got data engineers who maybe don't speak security, and you've got security engineering that you need to organizationally line up and make sure they're speaking. On top of that, you need to consider the business timeline and objectives. We've got pre-canned and pre-vetted tooling, which you can either take off the shelf, or if you've got custom needs, you can use it as a reference. And you want to think about your future plans. I love this analogy of the unfinished basement. I just have had the misfortune in life of working on products which also have the unfinished living room, unfinished kitchen, and the bedroom, which is actually just a tarp flapping in the wind.

Yeah, I love it.

I think that's a good summary. Yes, I think that makes sense to me. And I think it's more actionable for somebody who is in the planning stage. So these plans and this approach obviously make sense to me. So sometimes I know that there would be forces kind of beyond your control that actually force your decisions regarding security controls. So I am bringing up the compliance and regulations again. So are there situations where a regulation or a mandate would kind of force your decision regarding security controls or approaches, no matter what the threats are? I'm not talking about you deploying antivirus on your data warehouse because of PCI DSS. I'm talking about something a little bit more sane. How do you deal with situations where there's an externality that forces your decision regarding data warehouse security? And what can those be?

Yeah, thanks for bringing that up. I think one of the things to think about too, regulations are going to play definitely an important part of considerations. And to me, they sort of represent a min function for some of the security needs. By that, what I mean is I recall a customer sharing with me about a time in their career When PCI, for instance, hadn't yet recognized AES as a viable standard to use. And so, that just makes it interesting to think about things from that perspective, that instead of him having to use Triple DES instead of AES, he had to go through some of those sorts of discussions early on. Of course, that is not the case with PCI today. From a regulatory perspective and some of those controls that we need to think through, I'll just give a couple examples from some different frameworks. if you will. So from a privacy perspective, I think we wanted to try to avoid some of those discussion topics.

Uh-oh.

One of the things that I wanted to think through was we have an understanding of where the data needs to be, data regionalization needs that you have. From an insight perspective of DLP, we wanted to use some of those controls to make sure that we properly encrypt and de-identify some of that data, and we can use data catalog to help with Some of that information. If we take a look at some things from like FedRAMP, for instance, and maybe using NIST 800-53, we obviously don't have time to go through all 20 of those families, but maybe focusing on like access control families, if you will, that separation of duties and least privilege is a really important aspect of some of the things that we do. So you'll see that it's important to have things like column-level access controls to add additional context as to who is logging into your systems using Access Context Manager and some of those items as well. And then if you take a look at PCI and some of the data encryption data requirements that they have in place there, it's important to understand how to augment any of your key management capabilities with using it, protecting those keys with a FIPS-certified HSM. In our case, we've got CloudHSM to help with those things.

We get a lot of throwback references on the show. You might be the first person to bring up Triple DES. That's a blast from the past for us, for sure. I love this answer on controls. And listeners, if you haven't looked at some of the really innovative ACLing that's available in BigQuery, I can't believe I'm about to say this, but I actually really get excited about the ACLs and capabilities there in BigQuery. I want to switch gears a little bit, Erlend, or ask about how do we data governance in a cloud data warehouse?

Yeah, this is a really interesting aspect for me to think about because we have the chance to talk a lot about trust in the security world. To me, data governance is bringing about trust into the data world, if you will. So it's that trust in your data. It's the trust of your data's quality. integrity, its usability, and sort of the security aspect of all of those different things. When we think about data governance, from the inception of some of the things that we've been doing, data governance is, there's a lot of different principles that we've been able to apply from one of the books that some Googlers have helped write. It's called Data Governance: A Definitive Guide. One of the authors actually has helped shape some of the architecture early on for things like our data warehouse blueprint that we have here, where it just shows how we were able to bring collaboration across our different experts across our different products together to help think through what are the data governance needs that we have for a data warehouse. Some of the things that are practically, when you think about data governance, it's really about setting up some of the different policy tags that we have in place that allow you to identify different columns within your datasets and attach them to a taxonomy that you could define within things like Data Catalog. And so that taxonomy of how you you know, label your data. Maybe it's confidential data, private data, sensitive data, and public data. Whatever that taxonomy is that you have there, it's very important to understand how to tie that and how that relates to your data.

Hmm.

Once you have that mapping in place, you could then take that and apply additional security controls that we have, such as Cloud DLP and de-identification templates. So that way, any of those columns that maybe are marked as private have the appropriate transformation defined and it's being used for that particular column. And then you now know what DLP keys, what encryption is being done there and how that's being protected.

Oh, that's really interesting.

I know the data governance subject was kind of on the— makes some security professionals a little bit nervous. And I think that your answer is very crisp and very clear. And I kind of liked how you phrased it. I've seen a lot of data governance discussions kind of go in the BS territory very quickly. To switch gears from this, I don't know if you can accomplish this in the audio format, but when you think of the whole set of security controls that are there for the taking if you're implementing a data warehouse, of course, there are staples like IAM encryption, a few other things, but like, can you read the whole menu to us and hopefully not bore us in the process? So what controls are there to implement?

Anton, you can't ask somebody to read a menu on a podcast.

And not bore us.

Let me make it a little easier, maybe. What are the ones that are unique to cloud? And even better, what are the ones that are unique to Google Cloud? See, listeners, this is real time how a podcast gets made.

Well, I love it. So let me try to approach this a couple of different ways. So of course, we've got different layered controls that are built in, like some of the things we were talking about earlier. A lot of the capabilities and services that we have and are using, when we think about this data warehouse, are all about sort of like this serverless notion where there's not really infrastructure exposed to our customers to think through some of those things, whether it's using BigQuery or maybe it's a streaming engine for Dataflow and things like Data Catalog and Cloud DLP. When we think about those sorts of things, some of the unique things for both, and we'll kind of address them maybe for preventive controls and detective controls as well, but a couple of the unique things when I think of data governance, one of the things I enjoy has been that with Data Catalog, using the taxonomy that you can define and then apply that dynamically to some of the policy tags to set up those BigQuery controls that you were talking those ACL controls, talked about earlier about column-level access controls as well. When we think about elevation of privileges, we've got a lot of the ability to have fine-grained access applied down to the resource level, which I find very powerful. Sometimes instead of it being done at the project level or something higher level, you can actually identify which identities and bind those policies to the particular, like for instance, using keys, to that particular key. And then we have org policies in place as well that, for instance, we have the ability to disable service account key creation. So you can help really prevent at an org-wide setting, apply some of these broader controls and have them in place. When I think about some of those exfiltration controls and things that are unique to the cloud, I like to think about VPC service control perimeters and how we can define some of those trust boundaries, those bridges that we have in place, and marry that with some of the other org policies to prevent the ability to have external IP addresses. So then you can take some of our private restricted networking capabilities as well. So that way folks that are trying to go into the environment or use some of the APIs have to provide additional context and information with that. And then the last thing I think about too, from an information disclosure perspective that's unique, I think, is our Cloud DLP. de-identification and re-identification capabilities. It's been really good to separate those from our data flow pipeline. So you have one, you can think about how you can de-identify data and then re-identify data using a separate control flow that you then understand. And the other thing to think about too is establishing this kind of notion of a cryptographic boundary using our CloudHSM capabilities that have some automatic key rotation capabilities there. So. Hopefully that gives you a sense of some of those controls that we have in place that are unique within Google Cloud.

Perfect. Thank you very much. This is a really nice and extended answer. So we are about to wrap up and I wanted to ask you one final question. Any further reading for our audiences? Please share some links, some resources. We'll put them in the notes.

Yeah, absolutely. So there's a couple things. We've got our data warehouse blueprint that we can link to that. But something I was thinking about too is we've been talking about this journey and this transformation, if you will. There's a great book I'd recommend too. It's called Creativity, Inc. It talks a little bit about Pixar's journey. It gives you some tidbits ranging from various mental models when you're facing different challenges, cultural values that are important during a transformation, like candor and having brain trust, and then also practical leadership principles. And I think all of these things are important to think through when you're thinking about that digital transformation and how do you bring a data warehouse and secure that within your cloud environment or something. That might, you know, as we're thinking about journeys, a different type of book about journeys. And of course, we talked a little bit about The Data Governance: The Definitive Guide. That's always a good one as well.

Perfect. Thank you very much for this, and thank you very much for being on the show.

You bet.

And now we are at time. Thank you very much for listening, and of course, for subscribing. We hope this talking about secure data warehouses was way more fun, was as fun for you as it was for us. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify. We are giving a bit more attention to our Spotify listeners lately, and wherever you get your podcasts. Also, you can find us at our website cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. And of course, your hosts are also on Twitter, @anton_chuvakin and _timpeacock. Tweet at us, email us, argue with us. And if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security. Security.