EP233 Product Security Engineering at Google: Resilience and Security

Security running.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior PM for Detection, UEBA, Risk, and all kinds of good stuff in Google SecOps. And of course, Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on the Google Cloud Security Community page. Anton, we are talking about one of our favorite topics today, DNR, and we've got a couple of really interesting takes on the space.

Yes, correct. And I think it's also useful to remember that from the 220+ episodes we've done, the episodes that cover detection and response at Google are among the top— actually, top 5 and top 3, like the best episodes we've ever done so far. I mean, there's just no error.

Well, best as measured by listens. I don't know if that's best as measured by how much we love them. I love all our episodes equally.

Episodes. There is that, yes, except for that one.

Except for that one. There's that one. So, so's the so. What are some of the highlights today? Because I thought there were a couple of really interesting things around SRE, a new take, or at least a different take on the shared responsibility model than the one you usually get in the cloud security space. What else should listeners look forward to?

I think that SRE collab with security did come up in a few past episodes, but it's interesting how it's one of those good things to do, which people don't do outside of Google. I mean, I'm going to go on a very quick rant. People sometimes ask us, hey, what is your secret for doing certain thing well at Google? And we say, oh, it's X. And they say, we can never do X. Tell me another secret. And I'm like, yeah, no, actually, kind of X was a secret. And if you can't do X, then sorry, like, that's the answer. And I think that fits the SRE rotation with security teams. It's kind of in the same bucket. It's like, philosophically, they all agree, practically, they won't do it. And then they act surprised that we actually do it.

Well, listeners, with the image of Anton spilling our secrets and customers giving him surprised Pikachu face, let's turn things over to today's guest. Today we're joined by Christina Ventila, a Security Engineering Manager in Google Cloud's Office of the CISO. Christina, thank you for joining us today. We've done this How Google Does It series for some time now, and we've touched on threat detection, and we recently had Medr talking about threat modeling. Could you give us some insights into how product security engineering has evolved? You know, we've had a lot going on in this area. It's such a key area to all of our success here. So tell us about product security engineering, what's going on there?

So yeah, I'm part of the team that's called product security engineering for close to a year now. Meder, whom you've had on the podcast, is one of my colleagues. First of all, I want to say I'm very excited to be chatting with you folks. Both myself and a lot of my friends are huge fans, so very, very excited to be here. You're too kind. Completely gushing over you and the show. So very big fan here. Yeah. So, okay. To answer your question, Tim, product security engineering is part of the CISO organization. Me and my team, we are looking at the different Google Cloud products and areas, and our goal is to understand and improve the product security posture of the services that we, Google Cloud, as a company, put in front of our customers. So what does that mean? It's a bunch of work that connects end-to-end a lot of things. One of them, only some of them, are the threat detection and vulnerability research. And as I— just to kind of connect the dots here, I was part of the detection and response team in my previous role at Google. So that was also an amazing, amazing experience, learning from these folks who are like legends in this field. So I was in that role for a while and looking at building detection capabilities to improve the security of the Google services. And now I was very interested, and I had this opportunity to look at the GCP, Google Cloud Services, on a more end-to-end basis and kind of have the exposure to understand how the business leaders make decisions, position the security practices and best practices in this field. So... Kind of, this is what sold it for me. And I'm learning every day. Sometimes I joke with my team that I don't, I don't feel like I know how to do this job. But I think this is—

Don't worry. I joke that with my team about myself and about Anton. So it's all good.

Right. Yes. That's— I usually don't joke about it, but that's fine. I don't take it seriously.

I think it's a good gauge to continue learning and to be humble because when In this field, I think if you get too confident, I would be worried if I got too confident in my skills, given the space of security and cloud.

That's a fair point. I think that overconfidence led to all sorts of exciting Mandiant engagements in the end, to try to, you know, make a joke out of it. So, let's switch a little bit, tune the subject. And one thing that came up as very fun on both previous episodes, such as with Tim and with others, and of course, with our conversations with clients, Is this whole idea of applying SRE practices to DNR. And I would say that three four years talking about this to clients still produces occasionally confused effect because people just I'm sure system administrators before security are still too attached to this whole SOC with big screens, the analysts they click on alerts. So this mental model of what I call the knock DNA SOC is still very strong. So when you show up and say, "Hey, you should try doing the SRE." style things and SRE revolution and really change blah and blah, I get confused stares. So how do you think about it? How does Google balance this DNR with SRE thinking, if it makes sense?

So if I may unpack a little bit that question, there are several things. One, whether it's DNR or any other team in Google in security, you know, I don't know the details of each of the teams, right? I've been in 2 larger teams and collaborated with tens of others. So if you allow me, I will I will answer on how I see the engagement with SRE or how I see SRE kind of principles being applied. And there's books about this, you know, there's 2 SRE books from Google. There's also Security: Building Secure and Reliable Systems, also some years ago from Google.

Yes.

In many ways, why should thinking about a security system or a secure system be any different than building any system in a cloud environment, right? So just to give a few thoughts as to make it more concrete, right?

Mm-hmm.

When I'm thinking about SRE principles or SRE engagement on SRE mindset, it's not about that person sitting there in front of a monitor holding a pager, right? What we're talking about, we're talking about service level objectives. We're talking about error budget. We're talking about mature monitoring. We're talking about minimizing toil. We're talking about culture of blameless retrospective. We're talking about good, solid engineering practices, right? So, then where here did I say security? But what have I said that wouldn't apply to security, right? So, whether—

Well, wait, there would be people who say that engineering wouldn't apply to security because there are still people who want to consume security with products. And I know it sounds kind of completely crazy for us here, but We met people who don't want to engineer security. They want to buy security. They want to consume it. They want to click buttons. I mean, sorry for barging in with this, but it's kind of my— I just want you to produce good arguments for me to use with these people.

Okay. Let's try this, Anton. Let's try this. So, I've been in customer calls, and my colleagues in SRE, we are together in customer calls oftentimes. So I think if it's, I would decouple the accountability and responsibility from the, let's say, implementation or the responsibility of the implementation.

Mm-hmm.

So at the end of the day, the business owner in that very fluffy, high-level definition is the person who's running the business, right? Whether that's, I don't know, a cupcake factory or a Google Cloud Storage, right? There is a decision maker and an owner for running this business. So in a way, I see my team as providing a subject matter expertise and a support for this business owner, for this like very senior stakeholder to make decisions that will better position their service and their product. So we, as a company, we want to deliver the most secure cloud services for our customers, right? I can imagine a customer or an integrator would want to make a similar claim to their businesses, right?

Mm-hmm.

There's so much data, you guys know better than me, like ransomware and a whole bunch of mishaps with regards to security. So if we take this as a goal, that is good for business to have good security, then it follows that if I would have my own business, it's my responsibility. So, what I can do, I can delegate the execution of, let's say, a security program. I can delegate the execution of a pen test. I can delegate the execution of a security review or a threat model. But ultimately, I am accountable because security is just a support function to the business. And I think the best and the top leaders that I was very fortunate to learn from, and here, definitely in the past, several years since I've been in Google, are those who truly internalize that security risk is a business risk, and they see us as partners. So the moment we have— we come in with a conversation such as, do you have monitoring for your system? Have you enabled, kind of, let's say, my previous team, DNR, with sufficient telemetry so that they're able to build good detection so that, you know, if you get attacked or your service gets attacked, then—

Mm-hmm.

we can engage a detection and response and trigger an incident and do all that fun stuff.

Mm-hmm.

So we try to really meet our partners and our customers, whether it's my internal customers in Google or an external Google customer, and try to meet them where they are. And, you know, we're not saying, well, you should just use Google or you should do that. Just try to understand what your space is. What is your critical infrastructure? What are the main risks to the business? Do you understand the security posture of your service? Right? Are you— I don't know, I always give this goofy example. Are you running a cupcake factory and you're afraid that someone will tamper with the flour and the quality of your equipment? Will the equipment just stop in the middle and you can't deliver birthday cakes for Christina? Whether you do that or you run a global telco company, right? What are your critical infrastructure? Where are the attack surfaces? Do you have a threat model? Do you understand your main attack surfaces? What's the risk? Have you employed someone, whether it's Cristina's team or an external partner, to really understand this space, and then kind of make decisions that make sense for you as a business? And yeah, I don't see any difference in how I work with my stakeholders internally, whether it's the general manager of one of these major GCP areas, or a customer that I get in touch with, and we talk to them.

I think that makes sense. But it's also kind of interesting that it's obvious conceptually and not really practiced operationally by many people, right? At least that's my reaction to this.

Yeah. I mean, implementation kills strategy. It eats strategy for breakfast, right? And implementation is also dependent on each business and on each team. And even within Google Cloud, every team works slightly differently. And this is where we try to really apply our offering to what makes sense to each, even business unit within Google Cloud that we work with.

That is really interesting. And I'm still stuck thinking about a cupcake factory and how disappointed my customers would be if our machines stopped working in my cupcake factory. I don't want to work in a cupcake factory.

That sounds too stressful.

Can't disappoint children. So, I want to talk about data for a little bit. And how does our group decide, you know, what tools and what data is most important for our product DNR, and how are we dealing with that? And how are we dealing with all the data? What does this look like? What's the Google magic you can share with people?

I cannot share the Google magic.

Oh, no.

Also, I think, as that specific on the specific DNR space, probably folks whom you already had on the show can have a more up-to-date answer. But if I am to apply this to kind of my current role, and I'm looking at the product security engineering end-to-end, Right? When you say data, I don't know if that's what you mean, but I will state explicitly, like in an interview, like the way I interview folks on my team as well, like what I assume you mean, or what I read when you say that, is understanding the critical infrastructure and the crown jewels that are relevant for that business. And that can be, you know, the customer data that is stored on our platforms. It can be Also, the systems that need to be up to date or, like, have very little margin for error in order to support business-critical functions in, you know, healthcare, governments, transport agencies, and whatnot. So, what I'm thinking of when you say data is really understanding the critical infrastructure or crown jewels or however you want to call them. And this could be systems or it could be data. Or, like, if you add AI into the mix, the difference between like code and data kind of starts merging. So that entire ecosystem, how do we reason about it? And the answer is, or at least my perspective, we don't reason about it in isolation. We reason about it at security by talking to the people who know the customer need and the job to be done and the critical user journey that the data is going with. So it's not me or my team telling a business unit what their critical data is. It's that conversation and them explaining to us what's the critical user journey, what are the most important things for themselves to be able to operate and provide services to our Google Cloud customers.

And it's an interesting one. And I mean, I think that pattern we've seen in other Google, Googlers specifically, not just Google Podcasts, where something comes up that everybody conceptually thinks is correct and everybody outside of Google will think it's correct and everybody, every book on CISSP, whatever, from 1998 onwards would think it's correct, but it's just not being done much elsewhere. And when, remember the vulnerability management episode when we outlined how we kind of prioritize vulnerabilities by risk and do other things and both Tim and me had this reaction of like, Yeah, that's normal. That's what should be happening everywhere. And I think the magic is that it's actually happening here at scale, and it's actually a lot less popular in real life outside. Am I rambling too much?

I wouldn't say so. I think that's a very reasonable take on it.

I mean, there's still magic in some place.

Anton, what do you want to ask next?

I kind of wanted to drill down into this a little bit. I don't want to deal with But how do you talk to business and whatnot? But maybe how do you talk to SREs came up. You mentioned that our DNR team did not—

How do you talk to SREs? Whenever I talk to them, it's a mysterious encounter. So, how does that work?

Yeah, but maybe we should distill it to something more useful for the audience.

Sure, sure, sure.

It's fun for us even without distillation. So, you mentioned, of course, using DNR ideas for security, but you also mentioned in passing that you had to deal with actual SREs. And presumably in the case of detection response activities, right? Can you share maybe some ideas and what you've learned about how security teams should deal with SREs or DevOps or whoever?

Yeah, I think maybe the best example that comes to mind is the most recent, right? So my current team, we are, we have a working group and SRE is fully part of our process and the way we work. So there's colleagues who, when we do our job, like in our Normal day-to-day, even half an hour ago before meeting you, I was in a call and we had folks who work on the product, folks who work on more building security into the product, myself, where we have a more part of CISO, we have a more security assessment function. So we're kind of sitting outside of the product, but working collaboratively very closely with the product and the SRE leads in this space. And I'm talking about actually about the cloud AI products.

Mm-hmm.

And how those conversations go, first of all, all of us sit at the same table virtually, right? And when we are discussing whether there's a new feature coming up at Next.js, right? We had these conversations a few weeks ago. Where do we stand, right? What's the security posture? Do we understand the risk? Have we communicated the risks? And this can come from how we build and design the software. if we scanned it or we assessed it well enough, up to some level, there's no perfect 100% in terms of vulnerabilities. If we did due diligence with this and that, and SREs were like, well, if we look at the life of a Google Cloud AI model through production, right? What happens to it? So, it comes to be somewhere, then it has a journey up to, from the moment it starts in the concept pilot up to the moment it is deprecated. So, SREs are fully part of the lifecycle of the service and the product, and they're always involved, at least in my experience. And this is how I try to conduct myself. And, you know, just like security, they're engineers, they're very specialized on this other kind of skill set that's looking at monitoring reliability. They talk in different concepts, and that's good because we bring different perspectives. And I would talk to my team, you know, what are you concerned about? What are the things you care about? Where are the risks? What do you see? And there's no difference from talking to someone, to a colleague in SRE, except they don't use the words of vulnerability and exploit and attack surface. They think of failure domains and error budget and reliability and monitoring and observability. And so, the words are slightly different, but the concepts are very similar.

Could I ask an unscheduled question based on something you said in there? You were talking about working with product and eng. Do you have advice for product and eng leaders coming to the table for working with you? How should we show up to explain our products and be more effective partners? Hmm.

That's lovely.

Yeah. I'm still learning how to do that. I think the core mindset that I see in my management and my management chain and that I subscribe to is this idea of partnership. And what I hear is that traditionally security has been viewed as a blocker and a layer of features. And—

Security? No. Yeah, exactly. Exactly that.

Yes. And I mean, you know, in a way it's like, I fully buy this in, otherwise I would not be able to convince someone else that I'm, my team is genuinely trying to make the product better. And it's a bad day for everyone if we discover something a day before launch. So engaging us sooner can, at the very least, there's a reduced risk of mishaps and having a massive incident on our hands. very quickly before launch or at launch date. At the very best, or in the other desired state, is what I see our CEO of cloud, Thomas Kurian, mentioned, security being a key differentiator compared to your competition. And basically what we offer while, you know, keep learning and evolving our own product offering is, you know, what can we, my team as a security team, offer the business unit leader or the cupcake factory owner. And in terms of, in terms of, you know, assurance, maturity, understanding, coverage of risk. So preparing them to, you know, when they go on the market to their customers, a B2B2B business, they can say, well, you should use my Google Cloud service because, you know, it's mature. It has been reviewed. We have a mature operational setup. We can, you know, easily provide, you know, our reports around what has been tested, where are the risks. We have a mature process and understanding what can go wrong, right? Is it perfect? No, there's no such thing as perfect. But yeah, so far, people that I've worked with or pitched this value proposition of our team, they were super reasonable and understanding, and generally they wanted to learn more. I think it's also a matter of actually delivering, executing and delivering well, not just pitching this value proposition, but actually doing the work.

Googlers do, in my experience, tend to, but not always be, pretty reasonable and understanding people.

Yeah.

So, before we close, I still want to say that's another bit from the same pattern of how it should be done and everybody sort of knows, but we actually do it. I think that's interesting. So let's reflect on it later. Tim, you got to close.

Yeah. I hate to do this, Christina, because, well, I've been enjoying this conversation. I looked forward to it so much, and now we're at the end already. So one, do you have a tip to help people work better with security teams? And 2, do you have recommended reading for people which is not Anton's blog but could be anything other than that?

What I would do, what I've seen worked, is I would invite the folks from security almost like on a rotation. There's all goofy words for this program, and just to make it more exciting, I would invite someone from security to be on a rotation with the product, to be on call, or the other way around. It's a very personal thing, and it doesn't scale, right? Just out of the bat, as we talked about strategy and whatnot, but a very down-to-earth, very personal thing is just kind of get the security person to sit with you for like a week or 2 or whatever and do the opposite. And that's kind of very organic. It builds a lot of empathy. Like, when in my first role, I actually did a rotation with the SRE team and completely changed my mind and my bias and my stereotype. This was 8 years ago. So, I think while that's not a strategy that scales for large organizations, or maybe you can find a way to scale, I found that very powerful. Like, if you're a, you know, small-medium business, and you have a security team of, I don't know, 10 people or 20 people, overall, everything, vulnerability or not, just get them to sit next to the product team or the SRE for a month.

Yes.

I really like that. It's a very practical, very implementable tip.

And I think in terms of reading, I really like what Phil Venables posts. So, I follow—

Oh, now it's not just Anton's blog. Phil's blog has to go on the list.

Phil's blog as well.

Well, all right, listeners, you've got it. Phil's blog and spend some time working on the other teams. Christina, thank you so much for joining us today. Really appreciate you making the time late in the day where you are.

It was my pleasure. Thank you so much.

Listeners, I hope you enjoyed this as much as we did. And now we are at time. Thank you very much for listening and of course for subscribing. Please subscribe so you can get new episodes piping hot. And if you love our content, please drop us a review on your platform of choice.

You can find this podcast on YouTube, Apple Podcasts, Spotify, or wherever you get your podcasts. Also, you can find us on our website, cloud.withgoogle.com/cloudsecurity/podcast. You can argue with us on the Google Cloud Security Community site, googlecloudcommunity.com.

You can follow us on X, x.com/cloudsecpodcast. Tweet at us, email us, argue with us. And if we like or hate what we hear, we can invite you to the next episode.

See you on the next episode of the Cloud Security Podcast. By Google.

Security.