Security Operations, Reliability, and Securing Google with Heather Adkins

Security. Hi there. Welcome to the Cloud Security Podcast by Google. Thank you so much for joining us today. Your hosts here are myself, Tim Peacock, the product manager for threat detection here in Google Cloud, and Anton Chuvakin, a reformed and retired analyst and a member of the cloud security team here at Google. here at Google. You can find this podcast wherever podcasts are distributed, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you like our content, please do consider hitting that subscribe button. You can follow the show and argue with your hosts on Twitter as well at twitter.com/cloudsecpodcast. Couldn't be more excited about our guest today, Anton. You want to tell the audience a little bit about who's joining us and why it's so cool?

This one is going to be very fun. And while it may sound like it's based on a single presentation that Heather Adkins did at RSA 2021, it's really about more than that. We wanted to explore some of the lessons that Google learned in defending its infrastructure, in building security controls, in approaching security. We even will go into the question about how can we possibly revolutionize security. Of course, many of the sort of old hands in the cyber domain or information security domain would kind of cringe at every possible hint that something may change in a big way. But to be honest, Heather's presentation gives us some ideas about what may change so that security really does look different. To me, that's quite fascinating.

I also think, like, to the extent that people are cringing, we're in a world where if things don't change in meaningful ways, we're in deep trouble. It's not a happy place out there for defenders these days, is it?

That's true. That's true. And I think the debate on ransomware, for example, has shifted quite a bit because people like myself, frankly, used to assume that ransomware is kind of a poor people's problem, people with bad hygiene, bad basic security. But today, I very much understand that I was wrong when I thought that because people with decent defenses are hit by ransomware and the attackers just make an effort to bypass the defenses and they kind of have the way to Strike harder if the defenders push back. So to me, this is very fascinating, and we do need bigger changes to security.

I mean, the terrifying thing to me about the ransomware thing right now is you can see how much money these people are making, and one attack will net the adversaries in many cases more than the security budget for an entire year for even a mid to large size organization. There's an incredible asymmetry right now happening in the market.

Indeed, indeed. And while we can all dream that all of the money would go into buying Ferraris and maybe luxury apartments and taking their girls. friends to Maldives. That's all fair, but I am afraid that some of the money will go into the offensive research.

Oh, yeah.

So that to me is the point.

So what I love about the Heather episode though is, you know, we've got a guest here who's been defending Google nearly from the start of Google, and the techniques and ways of thinking and organizational design patterns that we're going to talk about on this episode I think have a lot of really transferable lessons to organizations at almost every size and scale. So perhaps with that, let's dive into the episode. Our guest today, so, so excited to have her join us, Heather Adkins. Her official title, aside from many others, is Senior Director of Information Security here at Google. Heather, our topic today is really about your RSA presentation. We previously discussed whether RSA indeed happened. We decided that the tree did fall, and we want to spend some time talking about what you covered. Your presentation, as I read it and understand it, you really had 3 pillars: zero trust, microservices, automation, and ZT production. Is that all you need to be secure and reliable in the modern world? Just 3 simple things?

Not at all. And first, let me say hello to the audience and really happy to be here. The RSA presentation is really grounded in this idea that we need to be doing things differently. And that's From developers to systems administrators, SREs, security people, etc. And I started thinking about why people do things the way they do today, and it's because that's the way we're taught. So I picked three examples of things that we're taught very early in our careers, whether you're setting up a new IT network for this first time or creating your first experimental web app or administering your first servers, because that's usually when you create your habits. And I think these are pretty bad habits. So I picked these three examples because I really think we need to go back to the fundamentals and change the way a we teach. teach practitioners, the many thousands of them who are being taught right now, but also for ourselves later on in our careers as well. That's why I picked these 3. That's not all you need to do in security, obviously. There's a lot more we have to do too.

I like that approach of taking 3 things that we kind of all understand or had as foundational and showing how you can change those and that that's the example of we need to change a lot of things.

But to kind of continue on this, and of course the other big theme of the presentation and of course of the book, which you'll mention a few times more, then Things should be secure and reliable. So when I read the book, and of course I've seen your presentation, deep in my heart, I still wanna question it. Like, I wanna question the connectivity between security and reliability. I keep thinking of examples like encryption, people losing keys and losing data, so it's kind of less reliable. So I'm striving to find a counterexample. By the way, I'm failing. But could you convince me that secure and reliable are in fact connected?

Well, I think in the news as we speak today, we've just seen a huge service outage at Fastly, which is a very large cloud provider, and they have come out and said that it was a service configuration error. So, let's just think about that error for a moment. We've had them at Google too. I talk about this in the book. Once upon a time, somebody configured our safe browsing product, left a space between a slash and a word, and it blocked everybody searching for every website. And somebody pushed that down into production. So, that was a reliability issue, right? I mean, it wasn't a security issue. It wasn't an insider issue. Somebody pushed a bad configuration. You could imagine at Fastly, perhaps, Somebody pushed a bad configuration and maybe nobody did a code review on that push, right? For example. Similarly, an attacker could break in and make that same change and that would be destructive to your infrastructure. So, when we talk about security and reliability, think of it from a solutions perspective. There are a lot of solutions that we could convince developers to put into place because it prevents them from having outages they have to tweet about. Similarly, it would prevent security incidents. from a hacker who hacked into your network. There are clearly things that exist uniquely in both veins, but from a solutions perspective, because we really need everyone working on security, we think there's a sweet spot between keep things up and keep things safe.

Is this the same concept that Washington, D.C. folks talk about under the banner of resilient systems?

I think there are some overlaps. I think they don't go far enough. I'd like to see us talking a lot more about what the military has pioneered in 2 key systems. You know, there's no single unilateral launch of nuclear missiles out of the United States, we hope. But similarly, why is there unilateral ability to push your production code or reboot your systems or do other things that may actually have follow-on effects that are quite severe? Similarly, there needs to be checks and balances. I give an example around zero-touch prod in the talk. If you're going to be submitting that configuration that accidentally has a space in it, why don't we have some pre-submit checks that look to make sure it's a safe check-in, right?

Okay.

So, there's other things that we can be doing around this. So, I like the government conversations. I think they need to go farther. I think we need to be much more realistic about doing things differently, making sure we're doing peer reviews everywhere, making sure we've got good automated checks, and more importantly, you know, Nobody realistically should be logging in as root anymore, right? I mean, we have enough instrumentation. We need to be building a lot of that so that the really risky things that we do every day in our production environments can be done much more safely. And then you save root for those really extreme times.

Break glass approach, right? Again, you save root.

Yeah.

I was at a startup for many years and, you know, the one account we had that was provisioned with org admin for our Google Cloud environment, the key for it was literally kept in a safe.

I'm glad it was in a safe and not somewhere else. But yeah, in the book, we actually have a safe as well. And we have multiple safes around the world where we keep things. Well, they just have to be physical. And, you know, one of the things we had kept in there was the smart card that would turn on the HSM that let the password management system at Google work. Well, that went down. And so, we had to get the backup card out of a safe in Australia, but we couldn't get into the safe. So, we had to, like, break open the safe, right? So, that system was really resilient, but not terribly reliable, right?

So there you go with your trade-off of security and reliability, Anton. We finally found it. It's safe.

Yeah, but you cheated a little bit and you kind of split resilient from reliable, which I guess you said secure and resilient but not reliable. Okay, fine. Yeah, I'll go with that. I think I like the line of thinking. I just want to test my own line of inquiry here. By the way, just as a quick side comment, one of my favorite parts of the book was that security as an emergent property. I just couldn't stop talking about it like for a while that the authors, I'm not sure who that was specifically, said that It's not about built-in or bolt-on. It's kind of what emerges once you build a system. And to me, that's really stuck in my head and sort of changed my life for the better. OK, maybe not. But it did change my thinking in some areas.

Yeah, and I think one interesting thing about the way applications are being built now than maybe they were 20 or 30 years ago is we would, 20 or 30 years ago, go and design something and spend 2 years building it and then launch it to millions of people, right? So that's how people got, say, Windows 3.1 or Windows 95. But today, we're building applications where every couple of days, I get a new Pokémon Go app, right? And it sort of updates behind the scenes. And that means that even as you are launching your service for the first time, maybe with just a trusted tester group or something like that, your emergent security properties happen much earlier. We have this issue inside Google where we do dogfooding, right? We try things out for ourselves for the first time. And the development teams are always really tempted to just say, we don't need security quite yet. It's just Googlers. Actually, you know, as soon as your system gets in the hands of users, you have to start thinking about these things. And because those development timelines are coming quite short now, it's almost from the very beginning that we have to be thinking about these.

It reminds me of the conference room in our building, Googlers Are People Too.

Googlers Are People Too, that's right.

I don't know if conference room names are allowed to be mentioned outside, but I think I can mention that. On the topic of modern app development, I think I wanna ask you a question where I ask you to predict the future, which is always fun for guests. You talk a lot about the LAMP stack in the RSA Prezone. I wanna know, when do you think it's gonna go away? I mean, we still have mainframes and COBOL. Where's the LAMP stack gonna make its last stand?

Well, I think LAMP as a very specific set of technologies, Linux, Apache, MySQL, and PHP, Perl, Python. I think all of those individually are going to be around for a very long time. They have a lot of utility. Typically, open source projects get replaced when there's something better, but there's always a long tail of strong advocates. I mean, lots of people using OpenBSD still, even though we don't see a lot of prevalent usage. And then I think architecturally, this idea that you have an open source operating system, an open source web server, database, and programming languages, like, that concept is never gonna go anywhere. We're gonna be stuck with it for a very, very long time. I think the idea that we need to be thinking about is how do we carefully construct that in a microservice kind of way? Because, you know, there's always gonna be small businesses who just wanna do something quickly. Unless there's a full replacement that's much easier, they're not gonna sort of go somewhere else. But then there's also a lot of legacy architecture. People have built their whole businesses applications on top of this idea, those are going to be around for 10, 20 years, right? It's not going anywhere. Although I'd love to get rid of PHP.

PHP, I think it's the evilest of the batch. No, no, no, there's absolutely no debate that in the LAMP stack, we are all gonna take our flamethrowers and try to burn the P.

Oh, I feel like ripping on PHP, that's almost punching down at this point.

Yeah, there is a little bit of that.

Yes.

So it's kind of interesting that I want to sort of shift gears a tiny bit from the kind of a classic approach, legacy approach that would survive for a while to sort of new And I sometimes agonize, back in my analyst days, I had to advise clients what to fear more, mistakes with legacy tech being brought to cloud or maybe mistakes with the new tech that people don't know enough. So you mentioned microservices, and of course, it's easy to do things securely with microservices, but only if you know what you're doing. So is there a risk that some of the novel stuff actually increases the attack surface, especially in less skilled hands? Feels like we're going to segment things, reduce the blast radius. There's all that. But won't there be additional points of compromise, again, especially when less skilled people run and build things? What's your take on that?

I think our approach is that, you know, we want to make things secure by default. So I'm gonna answer a different question first and then I'll come back. I think with cloud, we have a real opportunity in that a lot of organizations are looking at their IT setup and seeing how can we transform. And we've definitely seen this with COVID A lot of the hospitals are very keen to upgrade because they're having difficulty scaling or doing things very differently. We have companies who are to have their workforces work from home, right? So everyone's going through this IT transformation at this time. Because of that, we have a real opportunity to help them get things by default. And even if their staff are a little underskilled in particular areas, well, A, we should help them skill up, but B, we shouldn't make them get PhDs in security in order to get things right. I mean, that's just kind of silly. So I would say to everybody out there who is part of building these new cloud ecosystems, let's get security in there by default. Just make it a bit easier for everybody. A classic example I love is I'm using a code repo and the continuous integration and delivery pipelines from my cloud provider, and then I push my code into their cloud, et cetera, et cetera. All along that chain, we have the opportunity to do checks, right? Is this a vulnerable library you're using? Maybe you should upgrade that, or maybe just automatically pull it in. So, there are things like that that we can be much more helpful about. I think on the general question of are things that are new and different more risky, I think there's a risk, of course. I think with microservices in particular, it's risky if you're overcomplicating things. There's kind of a sweet balance of, you know, we've gone too micro, there's too many services to manage, too complex. We know complexity is more difficult to reason about, and if you can't reason about it, then you can't secure it. I give in the talk just a very simple explanation of an e-commerce site where you have, of course, the product database, A search functionality and a checkout functionality, right? So, that's sort of oversimplified, but it can be very tempting to take this kind of architecture and suddenly you have 1,000 services behind it instead of 3 or 4, right? So, I think study carefully how deep you're going and how complex you're making it, because then you can create more risk. But I think for the just basics getting into cloud, I do feel like that's an opportunity for people to Think in this different way. Zero-touch prod, starting to use safe proxies, starting to get beyond corp and these zero-trust ideas, 2FA everywhere, etc., etc. Take advantage of the fact that you're doing this transformation to transform security as well.

Yeah, I love the opportunity of cloud migration as like, don't waste an opportunity there. On the no-touch prod concept, and not just in a COVID-conscious no-touch way, aside from the national security upside of not launching the missiles without 2 people agreeing, what are the day-to-day security How does this really manifest for orgs when they move to it?

Yeah, I mentioned a little bit earlier, but maybe I'll just give an example to make it a bit more real. We do not encourage people to log in as root to the systems at Google. This is very different. My first day at Google, they gave me the root password, and I was suddenly upgrading SSH on thousands of machines. First day. It was kind of a wild experience. But we found this to be quite fragile because, of course, you make mistakes, or you didn't know all the machines you needed to do something on. Or there was a network outage and so you got 99 out of the 100 machines and then forgot about the 100th and then that one's got the bad config, right? So, what we did is we started to introduce these proxies and to understand the workflows that the SREs are doing. And then the proxies actually gate access to doing this thing. So, it will, for example, check the command line option I want to give, make sure it understands all the machines that I want to execute it on, and then it will go and do it for me and then keep an eye on it while I'm not there, right?

Mm-hmm.

blended automation with checking of commands to make sure it's accurate. But also, we just don't let people do some things, right? Like, you're not allowed to remove the root file system. And so in this way, we've been able to sort of help the SREs make fewer mistakes, which means google.com is up more often for people, which is great. And also, should an attacker take over their machine, it's going to be a little bit more challenging for them to laterally move into the production environment and/or execute something really dangerous on their behalf through the proxy. And you can couple that with multi-party authentication. So we can say you're on call, but look, this is a really dangerous action, so you need to get a second person to authorize it at the same time. Or you use a break glass, but then, you know, immediately that goes to a monitoring center and somebody calls you and says, was this you? Right. So I think it gives you just a good flexible framework for reducing both those reliability and security issues. It's all about resiliency, right? We're pairing these things on top of each other to make the system just a little bit more hardened, right? It's never going to be perfect.

I think a lot of security professionals make the mistake of looking for silver bullets when really it's kind of like driving a car safely. It's a collection of good habits.

But by the way, just as a quick aside to this one, how easy was it to sort of teach others to go there? Like beyond prod as a concept, which we sort of described in papers and of course we mentioned in the book and your presentation, it's a relatively new thing. And do you think it's adoptable? by basically non-Google type organizations. Like, this has been always my concern with giving advice to people in my analyst days. Like, I can give advice, but do they have a parser in their head— sorry for the log analysis metaphor— to really process the advice and act on it or not? Zero-touch prod, I have a fear that it's hard to put into practice if you're not Google. Is that true or not?

I don't think it is true, but I think you definitely have to approach it right, and you have to think about your org and the org you're in, the culture of your org. One thing that worked really well for us is, and we did face this resistance because I will admit that as a sysadmin, I don't want people taking root away. What we started to do was to create them anyway and offer it as an option. You know, hey, try it out. How do you like it? And actually we'd been able to build a lot of automation in there that made the job easier. So it was more tempting to use it, right? Like you don't have to enumerate all the machines where we're already pulling that out of a database for you. We're doing the double checking for you so you don't have to make sure you have typos, that sort of thing. And over time, it just becomes a bit easier to use. And then suddenly, you find actually they want to build features for it. And it sort of expands from there. And this was a journey, right? It wasn't something that we said, in Q3 of this year, we're going to roll this out, and that's it. It's small experiments. It's features. It's building momentum behind it. And that's how you change culture. You don't change culture by doing a tech talk, and then everybody walks out in agreement.

But it is a culture change, though. This means that 2/3 of IT people would be utterly lost. If you define it as a culture change problem, you're sort of playing into my argument, namely that it is hard. It's not about buying a new system or even enabling a new practice. It's kind of a culture change.

I think if you are in a traditional IT environment, let's just say a Windows environment, and you're managing Active Directory every day and you're RDPing in every day, it is going to be quite a significant change. And maybe that's not the sweet spot for you. Maybe the sweet spot is in your production web e-commerce environment and you start there first, right? Because If you take that down, it's not just the CEO that notices, it's millions of customers that notice as well. Maybe start with the most sensitive thing. The piece of infrastructure at Google that had many of these concepts first was our billing system.

Huh.

There, the controls were much tighter for a very, very long time. And then, over time, things grow. So, that's one way to look at it. If you're having trouble changing the culture of everyone, start with the most important thing because you're going to get a lot less pushback for the most important assets that people have general agreement that this is important on.

That rhymes with a theme that came up with both Max and John when talking about the Zero Trust migration of start with a limited surface, show users the benefit, build organizational trust, and then roll it out. You're really like tempting the users into it with like the golden apple rather than pushing them into it. I like it.

That makes sense. Zero Trust was very much culture change. And I think that in our other episodes with Phil Venables and his team, We had a lot of things related to cloud migration and how culture change is needed to migrate to the cloud securely. So, we may not get rid of culture change as an approach of security for the foreseeable future, it seems like.

Yep. And we did a whole chapter on culture. That's one reason why we did that is how do you get people to say yes? How do you get people comfortable with change? We talk a lot about this experimentation and sort of good usability and tempting them into it. And ultimately, they see the benefits as well.

So, the next question This is really my favorite question, and it's also a tough question. It's also a bit philosophical. It was more inspired by the presentation rather than like in the presentation. I think everybody agrees that SRE approach and DevOps kind of revolutionized IT. It really changed the very practice of IT management, how people do IT. Instead of like warring developers and ops, system admins, you know, stick pins into the, you know, little voodoo dolls to harm developers, whatever. It changed how IT is practiced at a deeper level. At least that's how I feel. Now, do we have anything of that sort that can actually change security in a similar way? I want to pretty much right away say that I don't think it's DevSecOps or DevOpsSec because first, nobody knows what it is. And if they do, they kind of see that it's tied to building applications, kind of an improved AppSec practice. But I'm talking about like full-on changes to security as big as SRE and DevOps or changes to IT.

Yeah. I also don't know what DevSecOps is. We intentionally didn't put a lot of concepts from that in the book, although you will see quite a lot of it, but not by name. I think actually where we're going is that a lot of the things we can do with automation and frameworks will get into the hands of developers. Maybe that's what DevSecOps is, right? When you get your Visual Studio or your Git environment by default, you know, we just partner the tooling with the safety. I mentioned the, you know, when you check in your code, if it's got vulnerable libraries kind of thing, right? Because actually, you don't need security engineers for that. It's not rocket science. You should upgrade your code. Don't run vulnerable code, right? Everybody knows that. And I think we can build zero-touch into cloud platforms to make it a little easier. I think we can do all of that, right? What we're going to need security people for is inventing all of that. And so, I think that's the revolution of security. We've been using these highly trained security people to look at log lines for a long time, but actually, what we need them to do is engineer the detection, and then putting that into the tools. I think that's where we're going, getting security people out of the business of the routine operations of these things, doing code reviews. I mean, we can automate code reviews in some ways now, right? With static analysis, fuzzing, et cetera. But we need people building the fuzzing tools, right? So, that's what you need the security people for. I think we also need security people to reason about really hard things like crypto. Like, you should not be writing your own crypto. You should leave that to the cryptographers. And then finally, I think we need security-minded people in every part of the business, right? I mean, if you're working in a public policy role, if you're working in a legal role, marketing, you know, every role that exists, you know, that we don't typically think of security being part of, we need security-minded people there as well. It's all part of the ecosystem. So, that's where I think we're going. I don't know if it's a revolution in security, but I think it's us pushing the basics To a more default state and then letting us focus on the really hard stuff.

Of which tool building is a big part because I feel like the revolutionary part is possibly that security and development sort of fuse a lot tighter, right? In many regards, not only for building secure apps, but also for building security apps, like your example of detection engineering. So whenever people discuss on social media, like, hey, do you need to code to be a security professional? It seems like in the new world, you kind of do, actually. So if you are training to be a security person in 2025, 2030, you probably need to be a coder. And this is a thing I would not share lightly.

You know, I listened to Julian Vehentz's podcast with you also, and he's definitely of this mind. I believe also that there's value in having security-minded people in every role. So when we talk about security people, I don't really know what we mean by that. But certainly, if your job is building tools, then yes, you're going to have to know how to code. In detection at Google, we've made the job of the detection engineers to build detection. And so, yes, they're coding. But, you know, I think public policy people who are helping us figure out what the compliance regimes should be, the standards, working with the government, we need them to understand security as well. And they're security people just as much as the people who code. So, I look more at what part of this puzzle you're playing to tell me whether or not you need to code. But everybody's welcome into the security tent. There's lots of work to do.

Yep. That makes sense. And thank you very much for this I think that I see a lot of graceful elements in your answer that I feel like are adoptable by others. So it may not be like revolution is in Q3 kind of thing, but it's very clear that security will look very different if these things are actually put into practice. So thank you very much for this.

Let's do this podcast again in 10 years and we'll see just how different we got it.

Oh, that'll be fun. Absolutely. You're on. I'll look for time on our calendars. I think 10 years out, it might not be so hard. We're getting close to the end of the episode, and I want to ask our traditional penultimate question, which is, what's your follow-up reading recommendation for the audience? Are there things aside from the book that you would recommend?

Yeah, I put some of this in the RSA talk. I really love the books coming out right now on containerization and microservices. There's one from Martin Fowler, Liz Rice, Richards and Ford. I think let's get people thinking about this new way of building applications, because I think that's going to be pretty revolutionary for everybody. So I would start there. Obviously, we've got the Google book, Building Secure and Reliable Systems. There's SRE books too. But I think if I were, you know, this audience probably listening, let's focus on getting containerization right.

Awesome.

Perfect. Please share one practical thing that people can do to be better at whatever the topic of our discussion. But today we kind of went very broad. So I don't know, Heather, maybe it's like share your favorite security advice. I don't know.

Let me say this. We want to take cloud to a secure by default state. We're still learning what that means. Everyone is. Don't forget your basic hygiene, please. Don't leave port 22 open and no password set on your instance. Don't install vulnerable WordPress. Beware of cryptocurrency mining. That seems to be the thing. But then as these better features, secure by default and things come down the line, don't be afraid to adopt them, try them out, experiment with them, because that's ultimately where you need to go. So I would just say, keep an open mind and ride the transformation with us.

And now we are at time. Thank you very much for listening, and of course, for subscribing as well. We will post links to the materials we discussed at the podcast site. You can find this podcast at Google Podcasts or wherever you get your podcasts. Also, you can find it at our website, cloud.withgoogle.com/cloudsecurity/podcast. And yes, it's a real website. You can go there. There are episodes. You can listen. You can see details. Please subscribe though, so that you don't miss episodes. You can follow us on Twitter as well, twitter.com/cloudsecpodcast. And of course, your hosts are at twitter.com/anton_chuvakin and twitter.com/_teampeacock. Tweet at us, email us, argue with us, and we like or hate what we hear. We can invite you to the next episode. It'll be fun. See you on the next Cloud Security Podcast episode.

Security. Security.