EP109 How Google Does Vulnerability Management: The Not So Secret Secrets!

Security running.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at GCP, and Anton Chuvakin, a reformed analyst and now senior staff in Google Cloud's Office of the CTO. You can find and subscribe to this podcast wherever you get your shows, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please hit your subscribe button. You can follow the show and argue with your hosts on Twitter as well, twitter.com/cloudsecpodcast. Anton, this is a fun episode. We have a guest joining us from apparently snowy Munich talking about one of the hardest problems in security, which is a good vulnerability remediation program. And I think it's especially interesting because she's the leader of it and she does it at Google scale. Imagine remediating vulnerabilities at Google scale. Just mind-bending.

Yes, but also imagine moving at a speed that is kind of characteristic of some of the really enlightened but small companies. So in that sense, in terms of, we've done a lot of episodes on how Google has certain magical superpowers in certain areas of security.

Like binary authorization. That's a magical superpower. There's some cool stuff, but this one, I think you and I were both surprised that there maybe wasn't magic. There was just being really good at your job.

Yes, correct. And it's not like we have this little tamed ML unicorn that just goes at night and patches all the systems. If you describe the pieces of our machinery for doing VM so well, The pieces are kind of the same as everywhere, but they happen to be working really well for a lot of reasons. Finely polished.

They're finely polished. They're well lubricated. They're free of gunk.

Yes.

And I think what's special is, you know, vulnerability program to me, I expect to be not the most exciting thing on the planet. But when you think about it from a mission standpoint, and I think this is a thing that a lot of my favorite Googlers have in common. is that the mission really matters. Remediating vulnerabilities at Google is really important because if we leave vulnerabilities unpatched, it's meaningful personal data at risk. And so this episode actually reminds me a lot of the Mike Sinnoh episode where there's a real mission-oriented aspect to it that makes the work meaningful and special. So perhaps with a couple of shoutouts to prior episodes, let's turn things over to today's guest. Today we are joined all the way from Europe by Anna Opeira, a staff security engineer, European lead of the Vulnerability Coordination Center here at Google. Anna, thank you so much for joining us today. And I want to just kick off by talking about the vulnerability management program you're a part of. So tell us about the scope. What does it cover? Are we talking operating systems? Are we talking off the shelf? Are we talking about custom code? All of the above? None of the above? What are we doing for vulnerability management here at Google?

Thank you for having me today, Tim and Anton. Greetings from Munich. Getting close to winter and snow is coming.

Nice.

But our Vulnerability Coordination Center is still having a great time at Google. So as you know, Google in its over 20 years of existence believes that a key part of keeping all our customers and our users safe is to understand our vulnerabilities and ensure that they're addressed consistently across the So we've been developing quite a bit of processes and technologies around that. And it's an area that we're really passionate about getting right. At the center of all of this is the Vulnerability Coordination Center. And this is where we are looking at making sure that threats posed by both software and hardware vulnerabilities are resolved and mitigated. Basically, we are looking at establishing cross-company processes around vulnerability management. We look at SLAs, service level agreements, going to touch a bit on that in a while. We are looking at handling and reporting processes, about roles and responsibilities, and many more. And want to highlight that there are many parts, many groups in the company that deal with vulnerabilities in one way, shape, or form. For example, probably many people know our bug bounty program, Vulnerability Rewards Program, where bug hunters are rewarded for finding vulnerabilities in Google software. We partner with them and with many other teams in the company in order to make sure that each of us understand what our role is, what our responsibilities are, and that we respond consistently to issues. We basically, in the Vulnerability Coordination Center, maintain situational awareness of around industry news on vulnerabilities, including zero days. And on the operational side, on the reactive side, we coordinate resolutions, the resolution of issues. So we work with product teams to fix defects as well as with PR and with support to make sure that our customers are informed of any steps we are taking or that they might need to take. We inform And respond to regulators who might have inquiries. And we work with Google's leadership to ensure that all the remediations and coordinations have appropriate resources set. Of course, Google is not the only company that exists in the industry that deals with vulnerabilities. So we partner not only internally with many different teams, but also with companies across the industry on coordinated vulnerability disclosures and ensuring that all of us are making the internet basically a better place. And last but not least, we try to also cover the proactive side and work again with our partners on that. And we do data by doing data analysis to find trends, classes of problems, so that engineering teams can take proactive measures to reduce risk. I know I haven't answered your question, so going back to that, basically the Vulnerability Coordination Center deals with all of the above.

Got it.

However—

That's a lot.

Yeah. The advantage of doing this at Google scale is that we actually do have processes in place to cover a lot of categories of vulnerabilities. So, we might have certain procedures to follow, particularly when embargoes come into play, when there's particular restrictions with regards to deadlines and what information can be shared. We have particular procedures for dealing with infrastructure vulnerabilities compared to hardware vulnerabilities.

Wait, so it includes firmware as well? If some of the hard drives we use have a firmware wound, it's also your job to fix it?

It's not our job to fix it, but it's our job to make sure that it is fixed. So you will not see myself or my colleagues go to our next data center and replace the hard drives. But we are partnering with the data center operations teams, for example, to ensure that they are aware of the issue. Either maybe we found out about it or they found out about it, that we have, if required— and here there will be a lot of ifs about what kinds of things the vulnerability coordination center will be involved in. If something is critical, we might want to make sure that the data center operations, when talking about firmware, follow certain procedures to make sure that either the systems are patched, or if that's not possible, that we have isolation in place, that we have detections in place and all of that. So.

Okay. So it sounds like given you mentioned critical versus not critical, and of course, Given the huge scope of this, I'm pretty sure that you use some prioritization process. And if I recall from the book, that would be Secure and Resilient Systems book, you mentioned that there's an impact assessment component. So, could you give us a broad framework about how you prioritize, how you decide what goes first? What's the impact assessment look like?

Sure. And there's not only one dimension when thinking about how do we prioritize and how do we identify. What systems to patch, if to patch at all, how many resources should Google invest in dealing with a particular vulnerability? There's several of them. So to call out a few higher-level categories, one of the first things that we look at is the severity. There's, I actually don't know an exact number, but probably in the order of at least thousands of vulnerabilities discovered every day. Each company and each team might only care about a small subset of them. Is a particular vulnerability being exploited? Is it critical in terms of particularly CVSS score that is a standard across the industry? And is it applicable to your particular systems? Because like if there's Patch Tuesday on Windows and your fleet is only based on Mac and Linux computers, you probably care much less about even the latest RCE on Windows. So there's the severity of the vulnerability is the first level, the first identifier that we will look into.

It sounds like there's no magic in terms of you don't have like a big AI in the sky that tells you what to do. Like, you really have to measure the same dimensions that a normal company would consider: severity, Popularity of an asset, resources available, right? It sounds like it brings back memories of my Gartner days when this is the advice I've given to clients. But I assumed you have some kind of magical system that tells you what to do, no?

I think that there's ChatGPT trending on Twitter right now. We haven't integrated it yet, but who knows?

Isn't there?

Who knows? Maybe it could cause good results.

I'm gonna go ask now while you're talking, by the way. You do know that, right?

Oh my goodness.

How to prioritize vulnerabilities.

Let me pick at this a little bit. That sounds really simple, and it sounds like all the basics every security team out there knows that they need to do. But what makes it hard to do at Google scale? What makes this at Google particularly challenging?

Like you said, the scale. There are, for example, in every piece of software particularly, but probably also on hardware, a very large amount of dependencies. You probably have heard that Google runs on a monorepo, on one repository that has many billions of lines of code, and that every team has access to. That means that our dependency tree is actually quite large.

Is it even a tree, or is it some horrible multiplicatively cyclic graph of horrors?

No, it's not. It actually comes with advantages to have this large monorepo.

Okay.

Because you can actually have, you have much more control with regards to your software and how you run it. So, the fact that you have one monorepo allows you, for example, to have code reviews. And with regards to dependencies, rules about visibility, what piece of software can be built by a certain a certain product. Are those allowed at all? You can have certain dependencies private so that only certain products or teams allow them. So, it actually does come with advantages, but only the amount of time to build a dependency graph and calculate everything and rebuild, it sometimes might remind you of the XKCD of, we're currently compiling, so can we get back to work?

We need that link for the show notes, by the way.

Yeah, we'll get that in there. You know, listeners, it's a good thing that I didn't end up going to law school because if I had, that question I just asked, that would have been called leading the witness. There's no way that was a fair question on a trial.

No.

I think one question then is, given our scale and given the scope and given all this, how do we decide what to prioritize and how do we decide how quickly a team needs to prioritize the remaining?

And I wanted to add to this because I want to emphasize the deciding and the speed needed. I recall again one story from Gartner days when somebody said, security team thinks 24 hours, but the database administrator thinks end of the quarter. Sometimes the gap may be like 2 orders of magnitude. So, how do we come to peace given our scale? How do we decide the speed of remediation?

So, in the impact assessment, we talked about severity, about dependencies, about how important vulnerability might be for our systems. And we have very shortly touched on embargoes and deadlines in the previous question. So taking all of those into account, we need to identify what to remediate, what systems are in scope, what is the relative risk level for each. So if we are to run through how mitigation actually happens, we first have a patch. We verify that it actually addresses the vulnerability. We roll it out. Ideally, we use the same testing framework that we have on for a regular release. However, what we want to accelerate is the timeline. So like Anton was saying, 24 hours versus a few days or a week. I think there are many products at Google that have rollouts that are supposed to actually last a week and not be finalized earlier because we do have billions of users and if 1% of our user population is affected by a bug or a vulnerability, that is a problem. So we need to make sure that when we do releases, we have plans for figuring out what to do when something goes wrong. So we will still go through our release framework and best practices, but we will try to do it on an accelerated pace. So Speed is something that should be taken into account when designing how your build pipelines are set up. Maybe you don't want to roll out to the whole user base, but you want to do the testing part, which should be expedited for the 24-hour release to 100% of your user base. You want to probably test on just 1% or 10% of that. And Afterwards, tune the release to go 100% after you're confident that at least a subset of your testing was successful.

Okay, that does make sense. And I think that it does mean that we can run very fast if we need to, in some cases. Like, if there's a type of an issue that needs to be fixed immediately, we can marshal resources and basically get it. And whether it's something like Shellshock or something else, Google's a big ship, but it can move. It can really move if need to.

And something I'd like to call out is that we try to communicate our actions. I was talking at the beginning with regards to the responsibilities of our team, that we work with executives to make sure that there's enough resources and there are enough resources allocated when doing vulnerability coordination. But we also work with our colleagues from press, from PR, and with support to communicate our actions and remediation plans to partners and customers. And since you mentioned Shellshock, I can tell you a little bit about what I know that Google did in this scenario. It was—

Ooh, please. People love stories about what Google did in particular scenarios, especially ones as public and big as Shellshock. So this should be good. What did we do?

This is, I must warn you, from stories since it was the year before I joined Google, or at least a few months before. So in 2014, Google Security learned about a publicly disclosed remotely exploitable vulnerability in Bash that trivially allowed RCE, code execution, on affected systems. The vulnerability disclosure was quickly followed by an exploit in the wild, as you might imagine. So the team assessed the risk. We were talking about how do we assess the risk? What indicators should we use for that? So let's look at what systems Google has. production. That's where the large or like a big umbrella of where Google's products, like if you go to google.com, like the search package will run on Google's data center in production. That's what we call our production system. So Google production servers, Google engineers identified as low risk because these servers could easily be patched with an automated rollout. There are a lot of processes already in place to deal with production or prod, how we in short call it. Once servers pass sufficient validation and testing, like I was mentioning, we do not roll out completely without any testing. Then we were confident to patch much faster than the usual weeks or days processes. Then another asset that runs Bash is represented by Google workstations. These were higher risk. However, they were also easy to patch quickly due to existing systems in controlling, in administering the fleet. We have Google-owned devices. People do not bring their own device to work. So this also enabled faster patching. The last category of assets, which was considered high risk, are Number of non-standard servers and inherited infrastructure. This category was more tricky to deal with because this did not have a particular process in place that we could automate and make sure that we can have an expedited framework and testing. So it had to follow a much more manual process, which required sending notifications, emails, messages, whatever contact information we had from the teams that run, or we knew had something with these particular non-standard servers and gave them very concrete follow-up actions for what to take. And this allowed us to scale the effort to multiple teams. In parallel to taking these 3 steps on these 3 categories of assets, the security team developed software to detect vulnerable systems within Google's network parameters. So don't forget about detection. That's an important part in vulnerability management as well.

So say more about that. Say more about detection and how that's been useful to us.

So it is important to not only think from the defender's perspective, particularly when doing risk analysis and threat modeling, but also to think about the attacker's perspective. So you want to make sure that you try to capture as many as possible of the attacker's potential actions. you cannot stop attackers from trying to do something. And it's always a bit of a cat and mouse game where we're trying to catch up with the latest techniques that the attackers come up with. But somehow we have, at least so far, been able to be ahead of the game. So whenever we notice that there are actions, that there's reconnaissance, that there's certain classes of attacks, we try to add signals. Again, detection is also similar to when you do an impact assessment, something that needs to be done not for everything, but taking into account prioritization, impact, and the different specifics to your organization. If your company does not contain an asset that is affected by a particular vulnerability, then there's no point in adding a signal for that until you actually add it. And with regards to how, how to do it, I'll let this for another story, but I do want to call out that gathering intelligence on threads is something that is probably useful to take into account. So monitor your system for port and application scans. keep track of DNS registrations of URLs similar to yours, buy threat intelligence data, or if your company has the maturity of requiring something like that, build a threat intelligence team to study and passively monitor the activities of known and likely threats to your infrastructure.

Yeah, always good advice. And what I love about this is in practice, you know, with my threat detection team, we see a lot of activity on a day-to-day basis. And one of the things we've gotten a lot better at over the past year is taking the activity we see and using it to work backwards into what are the other activity that's co-occurring or subsequently occurring that becomes another good signal to look for. And how do we then leverage that set of combined signals into a more holistic detection that we can then use for other good things? So we're getting basically to time, listeners, and I always hate to wrap up an episode, Especially one as deep on this. But given that we're at time, I have to ask our traditional closing questions. First, do you have a tip for improving vulnerability response programs? And 2, do you have recommended reading for our listeners?

A tip for how to improve the vulnerability management program?

Yeah, I know that's a broad question.

So, I guess the most important tip that I have is to build empathy and connections, build partnerships. I was mentioning that we are working with many groups at Google and we're working with partners across the industry. So you need to make sure that you understand what the roles and responsibilities from the different groups are. And ideally you should have talked to key stakeholders at times when it is not a crisis. If the first time that you go to a team that you need critical information from, at least in the security space, so the first time that you go to them is when you're dealing with a zero-day and there's a public exploit out there and you have, you don't know what time zone they're in.

Right.

Do they have 24/7 coverage? Do they reply to emails? Is somebody, Maybe this is a weekend and there's nobody out there until Monday, Sydney time, then you will probably have a lot of learnings to write down in your postmortem so that next time things will run more smoothly.

So, that kind of like pre-action bridge building, what's a good way to get organizations to do that? Should people be looking at doing, say, tabletop exercises, thought exercises, Should vulnerability program managers just be really social and walk around the office with a bottle of whiskey? Like, what's a good way to do that in practice?

I think RACI models are a good start to understand people's roles and responsibilities. Then once you agree, you have hopefully an open communication channel and you agree on your roles and responsibilities, tabletops or wheels of misfortune. Fortune or calling them different names.

Hold on. Wheels of Misfortune.

Wheels of Misfortune.

What is that?

SRE calls them Wheels of Misfortune. The same as tabletops. I think in security, people tend to call them tabletops and in SRE, such exercises where you go through a scenario that you prepare and then there is a dungeon master, somebody who runs the show and then there's—

That was me last night. I'm a dungeon master in real life, actually.

Literally.

Okay. Wow. And the person that is taking the exercise is a victim, and the rest of the team that might be contributing. In SRE, that's called Wheel of Misfortune, typically.

Oh, I love that. That's great. Well, okay. So if our listeners want to learn more about how to run a good Wheel of Misfortune exercise or how to get better at vulnerability program management overall, what would you recommend that they go out and read? Because I hate to leave listeners empty-handed, especially on a topic as important as this.

There's a large amount of literature out there. I must biasedly recommend the Building Secure Reliable Systems book since I'm— I was the lead of the project and one of the co-editors.

Amazing.

And I think there's a lot of public resources out there. You can find in the book also quite a bit of references. The next security community that maybe you're, you're part of might have tips and, uh, you might learn a lot from the next meetup, uh, that you participate in or the conference that you get to know somebody who is in maybe the same field as you are.

That makes a ton of sense. I always say that this is a field where it pays to have friends and it really pays to learn not only from your own mistakes, but the mistakes of others. So, Anna, with that, thank you so much for joining us today. Really appreciate having you on the show.

Thank you. It was a pleasure. Good luck.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikin and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Thank you, Ronnie.