EP189 How Google Does Security Programs at Scale: CISO Insights

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior PM for Risk Analytics and Cloud Threat in Google SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of the Cloud Security Podcast listeners on our Google Cloud Security Community page. Without any further ado, let's turn things over to today's guest.

And today we have Royal Hanson, the CISO of Alphabet. Thank you very much, uh, for coming up to our show.

Thanks, Anton. Thanks for having me.

So, I wanted to sort of start with an easy question that isn't about deep secrets of our infrastructure. It's just like, what were you thinking when you took the Google CISO job? I mean, this is just such a sort of warm-up question, admittedly, but also super interesting to me and probably to the audience.

No, it's a good one. And I think it actually was a big change for me. I'd spent most of my career on the East Coast. Certainly, I started in security at At Stake. Many of the people listening probably know At Stake from the loft and the early hacker teams that were part of that group. But I had spent most of my time inside of banks— Goldman Sachs, Morgan Stanley, Fidelity, a few others— really, I think, because as this industry matured, obviously banks needed to protect digital money, you know, in, in all of its shapes and forms, or personal data. And 6 years ago, I realized in talking with the folks at Google that so many of the same issues that were critical for banks as we digitized that industry were just exploding and emerging in every industry. And so, in some ways, it was, well, how do you help best as a cybersecurity professional? And like many people, I thought, ah, let's be part of the infrastructure. Let's be part of the foundation that everybody can benefit from— small businesses, medium-sized businesses, finance, transportation, etc. So, for me, it was just you couldn't turn down an opportunity to be a part of that story at that moment.

I think I've heard— it's funny. I mean, I'm sure there are connections deep in the ether between your journey and Phil's journey, of course, but I think I've heard the, this is where the infrastructure for the future is, kind of a theme, which to me struck me as kind of impressive. Like, that's where we are building the future. To me, that's not just glamorous, That's reality.

That's reality. And that's why I called Phil early on once I got— it was a couple of years before he arrived, but for exactly the same reasons. I'd worked with him at Goldman for many years on trying to build out safety and security. We worked with the Treasury Department, with the Bank of England. And we did things in that industry. But I think we were both attracted to helping the world globally in all industries.

Yeah, that makes sense. So let's get to kind of our lessons, because I think that our audience really loves hearing about, like, how Google does something, and they expect to have magical solutions, and everything is just like wave the wand and things just happen automatically across billions of servers. But in reality, there's a lot of work behind it. So our infrastructure is vast and complex, yet also modern, which I think is an advantage. So how does this influence the design of your security programs compared to others? I mean, banks or whatever other companies?

Yeah, it's a great question. And I don't think anybody can quite prepare you for the change from going from an enterprise of even a big bank to the scale of Google systems, whether that's the infrastructure or whether it's a product with a couple billion users. I mean, these are different orders of magnitude. And I think the thing I would say is you don't even have the opportunity, let alone the sort of organizational capability or the skill sets, to throw people at problems. You have to solve your security problems with design and, you know, secure by design and infrastructure that has security baked in.

Mm-hmm.

There's no way to scale a people-based operation in that sense. So we really invest, whether it's in software security, infrastructure security, think about vendor security, where is that injection point that we can bake security in, security by design. In fact, you think a lot of the things that we've seen this year from CISA and DHS and the Secure by Design pledge, that was what I saw on day one when I got to Google, which was we had baked into the web frameworks, into the mobile frameworks, into the infrastructure, the security libraries, input validation, all these, you know, the sort of typical things you hear talked about. But they were inevitable at Google in ways that every developer had to use them. And that, to me, was the big difference. And this had been happening for a decade or 2, as far as I could tell, when I got here.

I think the not throwing people at the problem came up in a lot of security operations detection and response topics when people just draw those lines about assets growing, threats growing, and then they assume that the SOC team in a more classic company has to grow at roughly the same rate. And we tell them sometimes that, hey, we decoupled the growth of threats and assets from the growth of the team. And people's minds are just blown. Like, they don't believe that. But it's the only way to run. Like, we can't run in any other way.

That's the difference, right? It's not like we made that decision. That was the only way you could do it. And in fact, you remind me, as I'm thinking about AI, because I'm sure we'll get into that, You know, we started with a small team that was experimenting on how to use AI back in 2011 with Max Poletto, someone you may have run into in the team. And we were experimenting with it in access control. Eventually, it became part of the detection and response capabilities.

Mm-hmm.

But we knew very early on we were going to need those kinds of tools if we were going to keep up with the scale. And that was true of detection response. Obviously, you've heard us talk about in Gmail. It's true of Gmail. It's true of the Play Store. And Heather Adkins, who you may have had on here, talks about this in terms of the cognitive offload. We are always looking for those cognitive offloads to help the people scale. And I think generative AI, and particularly in the detection and response— now you see it with Project Zero and some of the work they released recently on vulnerability finding. Obviously, access control, dynamic access control, some of the things we do in cloud and in Google, AI will be integral to that trend we started with, which is you cannot rely on people to scale this problem.

It sounds like from given the scale, and people assume that they would ask, yeah, what are the challenges running at this scale? But it sounds like it's the challenges but also advantages because certain winding painful roads are basically just closed to us. We can't say, well, we have money, we'll just hire another 500 people. No, first, it won't be enough. And second is then you have other problems. So, it sounds like, what are the advantages of running at scale? Like, that's probably a little bit of a weird question.

Yeah, I think, you know, one, it does force you to be much more disciplined, not just on the security or safety side, but in your overall architecture, right? You cannot have a globally available service the way we have to in many of these instances if there isn't standardization and sort of a reliability in the observability and the monitoring, and think of our SRE work. So, we— In fact, that book is such a great example of how the SRE, the spirit of our SRE work at scale, combined with the security work at scale, you know, you look closely in that book, a lot of it's the same stuff. If you standardize and you make consistent these, let's call them invariants, Or control points, everybody can rely on them, that aren't just security, they're operational, they're about availability at the same time.

I mean, the toil-fighting message from that book certainly changed my mind about a lot of detection response and sort of how to run it, how to run it at scale. I remember reading the book and thinking, toil, that's all we do in SOC. What are you talking about? How can we eliminate it? There won't be anything else left.

It is a really good point.

I guess that's the point.

Toil, doesn't get a name in a lot of enterprises the way it has. Like, it's one of our top vocabulary terms here in a way that I didn't feel it was as true in other places. And it's good for the security team to confront because otherwise, I think often there are many security teams, you know, I've been part of them along the way, who are not thinking toil as an equal consideration when they design and develop and deploy. So, that all comes back to this secure by design at a level that is much deeper than I think a lot of people think when they first hear that term.

Yes.

We'll get to secure by default in a moment, but I think, like, people do assume that it's just a configuration. There's a switch, yeah, on/off, and it's just set to off, and that's really it. Like, but there's a much deeper science behind it and that people sometimes don't touch. So, there are also other unique and interesting things, and I guess maybe more even, like, surprising things about our culture. So, I want to sort of ask, 2 things. One is about what has been most surprising about Google's culture, and then maybe the same about technology. So, sometimes people assume that it's the very heavily engineering culture, and of course, it is. But that's also, to me, is a bit too short of an answer. So, what have been the most surprising in terms of our culture?

Yeah. I think I touched a little bit on it in the beginning, which is that our equivalent of an application security or a software security team had not thought of the problem as an assessment problem, that they were gonna review every piece of software.

Mm-hmm.

They thought of it instead as a set of invariants that they wanted to guarantee that any web or mobile app could guarantee, could protect against. And so they were largely large systems design professionals, maybe more than they were classic hackers in that sense.

Mm-hmm.

And so, that was one big difference. And I think, you know, Google, to your point, we've had wonderful engineers around this place of all sort of shapes and sizes and histories, etc. What was surprising to me is how many people had made their way into the security team over the years from adjacencies in what I would consider just large system design. And You know, sometimes we can be a little bit religious in the security world. You know this.

Exceptionalism is like our popular, most popular sport.

And I think here, you know, because the engineering culture was so strong, security and engineering, security and SRE, just like we were talking about, it's a little less religious, let's say. And I feel, you know, it's not that we don't have our issues of these sort of dynamics, across teams or ladders or groups. But in general, we have been able to bring people from engineering expertise, scaled engineering expertise, into security in ways that I was surprised at how successful that had been at Google.

So, that means that there's something— it's almost like, I mean, it's not a very scientific point of presenting it, but like something in the air that changes it from a traditional security culture of like, Let's get a hammer, try to hit it a few times. I mean, they, of course, do a lot of really good red teaming, but the system design and large-scale system design and understanding of properties is kind of a big thing. I remember from the SRE book, actually, no, from SRS book, there was something, I think, from Heather, where she said that security is an emergent property of the system. That was another one that flipped the switch in my brain, like, oh, wait a second, that's a very different type of thinking. That's very not not spread outside of Google.

It's interesting, you know, I used to ride my bike with Luis Barroso, who sort of wrote the paper on the data center as a, you know, the computer as a data center. He passed away last year, but he was a great friend here. And when we would ride our bikes, I realized exactly what you just said, which is that, you know, he's very into the ecology and wildlife and preservation or sort of nature. And he would compare the complexity of these systems that we're all building and growing to the ecosystems in biology. And you couldn't guarantee everything, but you could pick out invariants. You could design a control that would operate consistently, even if it didn't, you know, it wasn't end-to-end. And I think that's what I started to think of as, okay, let's think in terms of invariants. Let's think in terms of control design. Because we're not going to have— no one at this scale can understand every last bit. But what you can do is you can treat control design as a first-class discipline. And that to me is where we brought these engineers into the system, is to say, yeah, this is just a different objective for the control design, but you've been designing controls for Spanner, for BigQuery, for years. That same expertise is valuable here, and in fact, for that very reason, because the complexity of these large systems is such that it requires that level of expertise.

So, this part addresses probably the most important part, namely kind of culture and thinking, but I'm sure that there was something also unique about our technology and not just scale. So, when you started to take a look at the realm in the early years and maybe how you kind of grown as a CISO, like, what was the most surprising thing about our tech stack?

Tech stack.

Yeah, our technology here.

Look, the simple answer is because the Borg ecosystem and the associated Protobufs or the RPC environment had simplified the infrastructure in a way that anyone here listening to this would think of just as a cloud. Like, you're getting the GCP or equivalent cloud experience, but that's how everybody wrote software here. There was no— there's no world at Google outside of maybe just the first year or two where people weren't writing to what we would call a cloud-native environment. They didn't have servers, they didn't have endpoints in the same way, didn't have databases. Like, that's probably the most— there isn't a team here called databases in the same way. It was called storage.

Storage.

Because you had to have scaled infrastructure that everybody could use, and you could abstract away the idea that you had a database or you had an application server. You had a job service, job management, you had table structures, you didn't have databases and servers. And to me, that was just like, you know, I think that's what— you know this better than I do, working in GCP, that is the spirit of where GCP came from. But it was true from day zero here with people like Luis and Urs and others.

So, I remember for some strange reason I read the How Google Works book, which is, of course, written in 2010. And I think they refer to the time when this was born as the dark past or not so dark past. But it's weird that in 2010, it was seen as the long time ago we did this thing. But so indeed, it was like Kind of what we would call cloud-native now, but 20 years before it was a thing.

That's exactly right. And that was very clearly the big difference. Now, again, huge advantages there because you can design controls at scale. You don't have to go secure each bespoke sort of cobbler's implementation. So, there were advantages to that from a security perspective, but that is the biggest difference when I arrived.

So, I want to touch— while I'm super tempted to ask about AI, I want to go a little bit more about the teams. A moment, and about metrics specifically. So, many, even large bank, well-funded teams still suffer from sort of team, maybe team degradation, burnout, especially as they scale up. So, I know we do have some sort of magic in terms of how we grow the teams and how come we suffer much less from burnout and a few other problems. So, what is the magic? Is it the metrics? Is it something else? Why does it work so well here?

And, you know, first of all, you know, we put our pants on, you know, same way everybody else does, in the same sense that, um, no magic here. It's not magic in that sense. But I think 2 things. One, we have drawn on the larger Google community, and I don't— so I think to a certain degree we've tried to make it a first-class engineering control design Not a dark corner of the org where only certain people can enter. So, we've tried to sort of broaden the pool and broaden the team, make everybody a part of this. It's a team sport, right? You hear that a lot. I think that's really true here. The other thing is we underestimate the role that the Aurora events in 2009 going into 2010, when the Chinese threat actors broke into a number of companies.

Mm-hmm.

And you saw Google visibly, sort of audibly, publicly talk about it. And then to make some really radical changes to our infrastructure, whether it was what ultimately became BeyondCorp and Zero Trust that you've talked about a lot.

Mm-hmm.

But that, you know, that came from the very top, from Sergey, from Larry, from the leadership of the company in ways that I think helped. It was a very Tough moment in many ways. But from that were born Project Zero and the team that took it at sort of a radical new approach. From that was born TAG, and ultimately, you know, that's sort of the threat analysis team. A lot of what you've talked about in the detection and response. But even Mandiant, I view as probably sort of the, you know, the on that same curve of people being attracted to the approach to security. And that again wasn't just the security team. That was the engineering and executive leadership of this company, of Google, in response to the toughest moment in our history on this front. And that brought people under the tent. People realized this is not a specialist's world. This was everyone's responsibility, and they took some pride in it. So I think those teams— people want to be a part of those teams, and I envy those teams myself. I think they're great.

So, it's also something that has not decayed since that time, which is very challenging. Again, I'm bringing back the DNR examples where people build something great and 5 years later, it's mediocre, and 7 years later, it's just not great at all. We kept this. So, I suspect there is also some magic, maybe related to metrics, maybe it's SLOs, maybe it's something else that made us not just create it and enjoy it, but also to keep it. How did we— so, what is it, like a democracy if you can keep it kind of thing? So, a great DNR program, great access control, if you can keep it. And we did keep it. So, how?

So, I think first, and we talked a little bit about it, the engineering culture, the by design. So, these were, you know, they never devolved into bean counting in sort of the classic sense. They remained first-class engineering problems that I think we could keep attracting people.

people to.

That became true of AI. I think, you know, with, you know, the AI wave was helpful when GCP started. It attracted people who wanted to solve those problems at scale for the rest of the industry. But you made me think as well that it's a— the leadership of the company remains committed to that. And so maybe there is some continuity still from that era that matters. And so this isn't just about the security team. It's about the company taking seriously that culture, taking seriously that obligation. Look, we had the events of 2016's election. So maybe it's the counterweight to being on the front lines, being attacked by the best attackers in the world. They're not going away. So we don't— we really don't have the luxury of letting it decay. is what I would say.

So, it's almost like, you know, they get me thinking too, sort of from the humorous and somewhat cynical, we take security seriously, to we take security culture seriously, is how you make it work. We take security seriously, in 2 years, you don't. We take security culture seriously, then in 10 years, you still do. Maybe I'm overprofound in this.

Well, now you're reminding me of another part of Google's culture, which I really admire and I think has been tricky to balance, but it's done very well, which is that bottoms-up engineering-led culture. Like, people can get stuff done here even if they're, you know, it's not a management mandate.

Mm-hmm.

Good ideas have currency here. They find their way. And so, I do really think, you know, it's a cliché, right? But culture eats strategy for breakfast. If you get the right people in these jobs, And with the kind of engineering skills we're talking about, they'll solve these problems in interesting ways. And we've embraced that. It's not something we've sort of shut down in any way.

Yes. And I think that shows because some of the stuff is very clearly preserved in some of the things that we found we didn't lose since that time. And I think the not losing part is also way more tricky than some people assume.

Just on any front, let alone security, right? Exactly.

Yeah, agreed. So, obligatory AI question. So, I want to start from a strange side. Maybe 3, 4 years ago, somebody was asking me about the DNR, SOC. And even back then, people assumed that we have some kind of a big brain in the sky that magically detects threats. And then we have to tell people, no, actually, a lot of this relies on very good engineering, very good threat research, and very rapid processing of threat signals into detections. So, it wasn't only or mostly about some kind of a mythical big brain in the sky. I mean, of course, now we are a lot more, you know, using AI in a lot more places, but like, how do you think about benefits and risks of AI and maybe specifically GenAI to our craft, the craft of security?

It's a good point because in the early days, you know, I was thinking 10 years ago, 7 years ago, we had to use AI. So, this is sort of the pre the deep learning maturation and the large language models. You needed good data, you needed labeled data, you needed to really kind of take on problems that were well defined, you know, well constructed. And that took labor in a different way. But so I think we used it, but used it as almost signal processing in very specific ways. What you're now seeing is the usage of— and this is that cognitive offload that Heather talks about— just for what I would call more generic use cases, summarizing incidents, asking basic questions that you would normally have had to go to multiple tools for, analyzing code. We work very closely with the team that does our Duet, our kind of our assistant for coding. Really, that can quickly become a software security tool or a migration tool for memory safety, things like this. So what's happened is we've gone from very specialized AI implementations into more general use cases. And it's back to that defender's dilemma. How do you make the defender who has to be right all the time, attacker just once? But really every defender and everything they do ought to be benefiting from these large language models. And that's what we're rapidly chipping it away at use case after use case after use case. And you don't have to go label a bunch of new data for that. You're just using the same language model with a bit of tuning. And that's on the one hand. The other thing, just while I have the mic on that, it's important to distinguish between using AI to power safety and security.

Mm-hmm.

And our secure AI work, which is the framework that takes advantage of everything we talked about, secure by design in the stack, Vertex in the case of GCP, Gemini models being, having, you know, safety filters and and tuning built in. That's the whole secure AI. And you saw last month, and I think they're talking about some of this at DEF CON and Black Hat this week, but, you know, one was the Coalition for Secure AI under the Oasis Foundation, which I think is gonna be an exciting place to work out some of these things as an industry. But then the DARPA challenge is what's being talked about in Las Vegas this week and using it to help, security more generally. So again, I would say using it. So it's important not to be the— I mean, cobbler's children sometimes doesn't resonate to this generation, but if your father makes shoes, those children don't have good shoes sometimes because they're so busy making shoes for other people. It's important in security we use AI to enable our defenders, not just secure it. So, to me, there are two sides to that coin.

Yes. And that's the statement we have to explain to clients very often because, yes, unfortunately, in many minds, this is squished together and the result is kind of between— from security use cases to securing AI. So, thank you for highlighting this. But I think that it's also something we very consciously treat as a tool, not as a big magic button. You push the button and it's done. It's very clear that we think about it as a tool that humans wield.

Well, to me, I think of it as the most democratizing tool yet, in that it makes every human better at whatever they do as they learn to use it.

Perfect. That actually makes sense. So, let me go to our traditional closing question, which is one tip, and in this case, one tip and one piece of reading. I mean, you can do more pieces of reading. As far as tips, we were tempted to just ask about a tip to a CISO who wants to run programs as well as You do. It's a very kind of like a maybe a little too centric, a little bit too hubrisy, but still.

It's, you know, so I would say 2 things. That's good to get the reading ones. You made me think about a number of things. But one, the world is digitizing. By that, I just mean like things we do day to day, companies, what they do day to day. Every corner of the earth is being automated and digitized. In incremental ways. So it's not always grand transformation. You know, I almost feel like that becomes a, you know, too much of a cliché. But meanwhile, lots and lots of our lives are digitizing. You know, the pandemic accelerated that. You could think about as one just, you know, high-profile example. And so I think it's important that we hook our wagons to those. And we don't think of ourselves as trying to, you know, back to this point of keeping a circle around everything with a layer of protection or control. Become a part of all this digitization and hook our wagon to it. Embed the security in each one of these new user journeys. So, to me, it's, again, it's that religious divide. Let's not have a religious divide between the security and the engineering, the business and security. But in fact, let's join forces. and make security about fraud protection, make security about safety, physical safety in the transportation world. You could think about all these examples. That's one. I'm trying to remember the book just to sort of finish you out here on this, but the risk book, Against the Gods: The Remarkable Story of Risk. It's funny, I gave it to my son who is a freshman in college, just finished his first year in college. Against the Gods: The Remarkable Story of Risk. And it goes back and talks even about the Shipping and sailing and how, you know, risk management was done there. And it's funny to see my son who works in computational biology, so thinks about risk in his own way, we're just fascinated by this. And I think if you can remind people in the security industry that they're part of a long, long tradition of sophisticated engineers, designers, whatever, explorers managing risk of a new world of new technology, that's super exciting, interesting, and it also has a long history to draw on of science, of control design, of metrics, of risk-taking, not just risk managing in that sense.

Thank you. This was hugely inspirational, especially the book recommendation because now I remember the book, but I don't think I've read it. So, now, at this point, it's on my reading list as well. So, thank you very much for being on the show. Hugely useful and kind of like what we expected and more.

Thank you for having me. It's great to be a small part of what goes on here.

And now we are at time. Thank you very much for listening and of course for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton underscore Chuvakin and underscore Tim Pico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode. Security.