EP220 Big Rewards for Cloud Security: Exploring the Google VRP

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here as ever are myself, Tim Peacock, the Senior PM for Risk Analytics in Google SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on the Google Cloud Security Community page. Anton, we are talking about vulnerability response, and I think it's very interesting what you were just saying before we hit the record button. about the inevitability of vulnerabilities.

Yes. And I think that it's almost like the thought only came to me after we recorded the episode, is that people who run VRP programs not only know that vulnerabilities exist, they know that there are many of them. They're inevitable. If you dig a little bit, you'll find one. You can support your habit for drinking beer or your family at times with finding them and reporting them to somebody who would pay for them and then fix them. So, like, this is very normal. And when I remember my analyst days, I remember getting an inquiry from somebody, and I'm going to, of course, change enough details so it's not a leak of IP.

Of course.

That somebody said, my regulator demanded that I fix all the vulnerabilities.

All of them.

And initially, I thought they're pulling a fast one on me because, like, well, what do you mean? System responds to pings, some scanners identify as a vuln. So like, why would you regulate? Like, I wasn't really sure they're serious. But they said, no, no, no, we are serious. Our mandate is to fix all vulns. And my reaction was to scream, you're insane. That's not a thing. But it's interesting, like dramatic and weird contrast that I bet today there are people out there who think that if they just try hard enough, they're going to fix the vulns.

I mean, it's very easy to fix all the vulns. You unplug the server, encase it in concrete, and chuck it off a boat into the Mariana Trench.

Well, if it falls on your foot, you're still vulnerable. So you have to wear your steel shoes.

Make sure you wear your steel-toed boots.

Yeah, otherwise you're vulnerable and you still have bones.

At least until it's off the deck of the boat.

So maybe listeners, if you want a more optimistic solution to vulnerabilities than dropping servers into the ocean, let's listen to what our guests have to say. And with that, listeners, I'm delighted to introduce today's guests. We have 2 today. We're joined by Michael Cote, the Cloud VRP Lead for Google Cloud, And Adarsh, a security engineer here at Google Cloud. Vulnerability response is hard in general, and things are hard at Google Cloud scale. And so how do you handle this hard, hard problem of vulnerability response for GCP? How do you triage? How do we look at the right stuff first? How does this work?

So that's a great question.

Thank you.

7 of them.

Yeah, that was like 8 questions in one. So pick your poison.

I think the real answer there is that we do it a lot with collaboration. So we don't make any decision by ourselves. So something comes in, we immediately triage to determine what needs to be reproduced first. So once we've done that, we kind of go in priority order, also with the identity— the idea of how long something's been waiting.

Mm-hmm.

And then we reproduce every single issue to assess the real impact. Once that's been done, our first-level triage works with our second-level triage, which Adarsh is also a part of. To kind of pressure test their findings and their assessment before we then work with the product teams. So ultimately, it's about finding and then fixing the vulnerabilities. So then we work with the product teams, they do another assessment, and then finally, it goes— once we've filed the bug with the product teams, we then again work as a team with a rewards panel to further assess the actual impact to reward, and we always have to come to consensus for that.

That makes sense.

But wait, so I get that answer, but there's still a question of like exact monetary number, right? So you can have a committee meeting and talk about impact, but then you have to pay somebody. So how do you determine how much to pay? And in the tradition of asking multiple questions, as Tim just wisely did, what's the largest we paid and what it was for?

Ooh, good question.

I wanted to highlight that while we are reproducing the issue, we do try to find additional impact so we can like identify it. And during the panel discussion, we come out with like whatever full impact the issue could have had, even if the researcher did not go ahead and submit the impact. We do award the researcher for the maximum impact.

And how do we end up with a number then?

Yeah, how do you pick the number?

Yeah, it's like impact I get, but like here's the $37K versus $73K. How do we decide on that?

We have a very specific set of rules that are published for our researcher community. So if you go to the bughunters.google.com website and look at Cloud VRP rules, it basically says— so there's 2 pieces of it. There is the— and we call it the tier— the importance of how we see the product. So different tiering. And then we also have the impact that that specific vulnerability has. So, and it's basically a cross-product of those. So if you are in a Tier 1 product and you find a very severe vulnerability, then you're going to get paid more than that same vulnerability in a Tier 3. But the way that it really happens is we work with the product team. Everyone on the rewards panel will work with the product team to understand the exact impact And the maximum impact, like Adar said, of the report so that we can pay them out. And we go back and forth and we have to come to 100% consensus before actually rewarding. As for our highest, this actually happened a couple of weeks ago. There was an RCE in Borg, for those who don't know. Yeah, exactly. Which is like Google production.

Is this for reals?

Really?

Yes.

They found through our cloud product I won't mention which one. They were able to find an RCE in Google production. And for that, the reward was $50,000, but they also had a 50% bonus for our exceptional report criteria. And so they got an additional $25,000 for a total of $75,000 for our reward. And that was like voted on last week and they found it a couple weeks ago.

Wow. So listeners, you could Find a remote code execution in Google's production infra and get $75 grand for it. That's not bad.

Not bad at all.

But also, is that not kind of a pittance compared to what, like, a nation state would pay for it?

So that is a possibility. We tend to work with the white hat hackers that like to improve the overall security of Google and improve the security of our customers. So I've never been privy to anything like that happening.

Okay. Well, moving right along from pricing and the evil thoughts spinning in my head, maybe let's switch gears a bit and ask, where do we see the most submissions and what drives where we see submissions? Is it things that are new? Is it things that are popular? What gets submitted?

Let me, before you answer, and I think it gets political very quickly because there would be people who assume that the most submissions, the the worse the security is. And we know that not to be the case because there are many other dimensions as to why something gets more submissions. It is not necessarily that the product is bad, right?

Useful?

That is 100% correct. So our Tier 1 product—

That pause really almost killed me. You had this pause, and I'm like, he's going to tear me apart.

No, the Tier 1 products, just because we pay more by default, get a lot more. I will say some of our most popular ones are— Looker and Apigee and Vertex AI. I will say Vertex AI is not because of the poor security posture, but because I think everyone is interested, both security researchers and Google, in our AI products at this time. So I think just the popularity of some of our products and the general use of them gets more submissions.

That makes sense.

So it's more of a product popularity But I mean, wait, surely it's— now I want to backtrack my previous comment. Like, surely it's a combination, right? Like, if the product is very popular, but we gave it our all and made it the best possible ever on the planet, it probably won't get that many, right? So it's— isn't there some kind of a weird Venn diagram? Maybe?

It's probably true, but that's very hard to measure. So I would say none of our— like, as much as I would love to say certain products have Perfect security posture. I don't know that I can actually say. I can say though, like, we've recognized trends for more popular products, but I do not think that's because of poor security posture rather than actual interest in the product.

So one more factor I think is something to consider is how frequent we do releases on these products. For Vertex AI, we are releasing new features constantly. So that interests researchers to go on Vertex AI Search or Vertex AI because you have new features, new security bugs can come out of those features. So yeah.

Oh, that's an excellent point. And I think that that is not something I thought about, but it's obvious in retrospect. If we write more code, we probably write more bugs. And there's more motivation to look for those bugs. Tim is questioning with a cringe face. I don't know.

I think that there could also be an argument that if you're releasing code more frequently, you're better at releasing code and you don't have stale code waiting to come out. So, I would imagine there's some happy medium between I build my app once a year and shit is stale when it comes out versus I release my code every night and I have insufficient testing.

But to Atarsha's larger point too, though, the more frequent releases get more attention.

Oh, it's also that.

And so, that increase in attention is also very likely to get you an increase in reports.

That makes sense.

Hmm.

So, okay. Another sort of a bit of a context switch. So, if somebody writes us and says, I found a horrible bug in product X, and we say, no money for you because there are rejection reasons. So, what are some of the likely rejection reasons? Would you reject something for like bad spelling or not?

I love this question because my goal in the Cloud VRP is to give out as much money as we possibly can. I would love if every single report that we got was, Exceptional and a valid vulnerability. That said, and I'm gonna change your question from rejection reason to not paid. Ah, those are different. They are different because we pay not by the vulnerability you find, and actually I just had a more in-depth conversation with this, the researcher, recently, but by impact. So I would love for you to imagine that you've Find a privilege escalation vulnerability within a Google Cloud product, but you can't really do anything with that.

Oh.

So we're not gonna pay because we pay by the impact. Is it read? Is it write? Is it project?

Is it—

do you get into Google production? Those are the kind of things we pay. We don't pay for the presence of the vulnerability itself, which is somewhat confusing, and I love making this point. The other reason that we don't pay, and they're kind of hand in hand, is some of our researchers don't really think through who would benefit from this. So sometimes they might find something, but again, this talks to impact. We really like the good reports talk about, hey, this is our attack scenario.

Mm-hmm.

This is who's going to benefit from it and how they're going to benefit from it. And if you can't answer those questions, it frequently isn't going to get either rewarded or rewarded at a higher level.

That makes sense.

It sort of does, but also it's a little unclear. Like, can you give an example of maybe a privilege escalation where I can become an admin of something from a normal user, but nobody can ever benefit from that?

And I don't want to give specific—

there have been—

Make it up. Like, I don't need a real example. Just like, give me a hypothetical example.

Yeah, there could be privilege escalations that lead to— imagine you get a role, but it doesn't really— and it has, or maybe a very slight increase in the privileges. So you already, that previous role, you can use lateral movement to get to a new role, but it doesn't materially increase the amount that you can read or that you can write or impact a project.

Mm-hmm.

Imagine you stay in that same project with almost the same permissions as you did before. That would be an example.

That makes a ton of sense.

Okay, so because my mind is poisoned by Windows and I'm thinking regular user to admin, but this is cloud and it may mean that it would be like from one role to another with slightly more privilege. I think that makes sense. That puts it to rest.

So let me ask the flip side of that question, which is what makes for an exceptional report? You know, what, what's an awesome report? I mean, obviously RCE and Borg, that's pretty exceptional. What else counts as exceptional? How do I, as a researcher, give you a report that you go, wow, I love this report. I want to pay this person so much money today?

So we have guidelines internally and externally on how to make a good external or exceptional report. So some of them are clear steps to reproduce, how the issue is impacting, what is For example, you said privilege escalation. What privileges are you able to assume? What are your permissions associated with it? Then what would be the right mitigation to go ahead? When you document all this stuff, I think it makes a great report.

So basically, I should be really clear about this is the impact, this is how to do it, and this is why it's bad. It seems pretty straightforward.

And you can even go ahead and explain, like, how this issue is working. So we have, in some reports, we have seen that the researchers take that extra effort to explain how the security vulnerability is happening, or like they go through the chain of exploits to figure out the exact part where it is getting exploited. And that helps a lot.

That actually makes sense. And I think that I get the impact and somebody benefiting to me is just stuck in my head now as ideas that drive this because it's not like Oh my God, I found cross-site scripting in something that's only exposed to admin or something. Like, I can imagine some no-impact stuff. But if you just write it well, describe all that coherently, then it becomes exceptional. I love that. So, very much switching gears to something philosophical. And I feel like we may spend another 6 hours discussing this, but ultimately, VRP programs like this one for cloud are meant to improve security, right?

Yeah.

There are plus-minus 3 cloud providers, and the cloud providers, I feel like, are better than many other organizations at security. So, how are we becoming more secure because of ERP? I mean, maybe that's a weird question. Like, how is that making Google Cloud more secure?

So, we no longer have at least one RCE on board.

Well, that's an excellent point.

So, the whole purpose is to find and fix. vulnerabilities. I would say that, and I was just looking it up beforehand, we had about 400 reports in about 6 months last year. So we launched a soft launch, internal launch, in July of last year. October, we launched publicly, but we were behind the scenes since July and had over 400 reports. We had about 200 of those, about half, I guess, were legitimate vulnerabilities within cloud that we never would have found, that aren't hypothetical. And, you know, a lot of CVEs that might be found by scanners, they're more hypothetical because you could have mitigations in place. But for these VRP reports that are, again, proven that we've gone through 2 levels of triage, we have all these reward panels also looking at the product team, they are proven externally. exploitable vulnerabilities that can impact our customers and Google. There's 200 vulnerabilities that would have been in place. And again, you think about the RCE and BORG and getting root in cloud based on that to multiple products that aren't available anymore because of the existence of the BRP.

Yeah. So something I want to highlight is that, like, finding these issues requires a lot of time. And the researchers being able to dedicate that time is what lets them find these issues. Most of the areas where, like, internally, when things are getting rolled out fast, we security engineers may not be able to invest the same kind of time which the researchers are able to invest. And that's part of the reason why we are able to find such high-quality reports, and these issues help protect the customers.

This makes sense. Do you think it's better for the planet that there are 3 major compute substrates? Azure, AWS, Google?

I think you're pushing it way too philosophical, way straight to space, right? Skip the stratosphere, right?

I mean, to the moon, maybe.

I would love to answer that, actually.

Yeah, please. You see where I'm going with the question, right?

Yeah, my last company, I won't mention that because I'm not sure if I'm allowed to, but we actually went from on-prem to a multi-cloud approach while I started it while I was there. And I will say the security expertise that is within Google and Amazon and these, like, you have these certificates of attestation that you are secure and can work with these different government entities and regulatory entities, that the fact that you have specialists for all of these, where you have very generalists, I think, without these cloud providers, it 100% benefits everyone.

Hmm.

Even though it leads to a triculture or a monoculture, depending on— you think it's still better?

I think that the scale that you get, so like both from like an ecological perspective, I guess, and from the improvement of security posture of our products, because Individuals, and I used to write apps on the side. I guarantee you that if I had to implement that by myself, I would have done a much, much worse job than the Google security engineers would.

That makes sense. I could see that. So I'll bring us back to earth, and then I think we're going to get close to wrapping up. In college, I had this roommate who would— he was like a genius or idiot savant for finding web app vulnerabilities. He used to take us drinking, and he was on the Google Hall of Fame for web app vulns for a while. How much money are people making off of this? Is it more than just beer money these days? And do we keep like a Google Cloud Hall of Fame for vulnerability submissions as well?

I will say we had a bug swat event, so like a private research event in Malaga last year.

Oh, nice.

I got to work with someone. He was the MVP, like most valuable— MVH, most valuable hacker at that event. And he had quit his job, his full-time job, Like the 6 months prior, and this is what he does full-time with a wife and kids.

No way.

But not just with us, with many VRPs, right?

I would assume. Okay, so I asked him that. He actually just does it with us now because we pay on average the most. And one of his primary considerations was how quickly we pay and how quickly we triage.

Oh, sure.

And we do that at a much better rate than our peers, at least according to him. So I know it's much more than beer money right now and pretty exciting to be a part of.

Not only is it more than beer money, it also sounds like Google has 2 advantages here. We pay more and we pay faster. And I think if you're an independent researcher, those 2 things together, that's got to be huge. So that's awesome.

I think that alone pays for your presence on the podcast, right? Because it's a freaking ad for the program and now everybody would sign up and they earn either beer money or You know, living with wife and kids money.

And the more the merrier. So, I invite everyone to submit a bug and find out.

I love it.

And we also invite them for events. So, you get, like, free travel and accommodation.

And Malaga, for God's sakes. Yeah, I want that too. So, I got a weird one, and it's very, like, it's a very Anton question because I've been struggling with this for some time. I've met people who think CVEs in the cloud flat out don't work. And I've met people who think CVEs in the cloud are fine. So, how do we uniquely identify vulns? Do we like to file CVEs? Do we have to file CVEs? I mean, I see the logic behind not liking CVEs because if cloud changes very rapidly, then it's not the same as CVE in the Microsoft SQL Server version 2008, which everybody knows what it means. But CVE in something that has code changes changing every week, I don't know if it makes sense. So, how do we uniquely identify vulns, and does CVE work or not? And what work means?

So, I think CVE is a good first step. But again, I think we talked about this a little bit ago. CVEs, there can be a lot of mitigations in place in a company like Google where you have a CVE in a specific package that isn't used or it's not reachable, or there's other mitigations in place that significantly mitigate the risk of that CVE. VRP does issue CVEs, actually. So we started this about a month ago. Someone on our team, Phil, is in charge of the program for critical vulnerabilities in cloud—

critical.

We now at Google Cloud issue CVEs for all of these.

Hmm.

VRP is in the unique situation because of our researchers, as Adarsh was saying, they have this time that CVEs aren't really the way they go. They do exploit chains, and they're not usually using CVEs to find impact. I'm not going to just call it a vulnerability because it's usually a multitude of vulnerabilities and exploit chaining that they might use to actually get the impact. So I think that's a little bit of the difference between how our researchers are using CVEs and vulnerabilities comparatively to just, like, the CVE.

Did I understand you correctly? You said that we are publishing CVEs for the vulnerabilities that we patch in our cloud?

That is correct.

So that's interesting. Could you say more about that? Because I've heard traditionally, oh, there's no need for CVEs in the cloud because CVEs are only useful to help you patch. And since you don't have to patch because it's cloud, we don't need CVEs.

That's a strange argument.

Yes, this is a recent change. Goes all the way up to TK and Brad that now for any critical vulnerability, that we find within Google Cloud, we do publish a CVE. And it's not just about— and I will say it's more than just about the publishing, because sometimes they don't even need to take action. But it's allowing our customers to make informed decisions.

Hmm.

Also to take a look at their own environments, and maybe they see similar things. So it's not necessarily just about the publishing and, A lot of times they don't need to take action, but I do think it's about like just generally increasing the security posture for everybody.

Can I ask one more follow-up on this? Did the other 2 unnamed blue and yellow clouds do that?

One of them I know does. Shortly around the same time that where we published, they also determined to launch for critical vulnerabilities. I am not sure about Got it.

Interesting.

Yeah, there's one more question I just feel like I need to stick in there. I could tell.

I could tell you had that look on your face of one more.

Yeah, I'm turning in my seat and thinking. So, it's a spin on the question we already asked, and I think there is an obvious answer. Like, does VRP also reduce the risk of customers, right? Like, if somebody's a cloud user, obviously, the answer is yes because I use a service and the service just got the vuln fixed. That route is clear to me, but are there more risk reduction benefits for clients?

Yeah. So, there are times when the customers have reported issues to us, too, like our security researchers who went to customers finding issues in their environment, and then they got back to us saying that this is an issue with some of your products. Or maybe they're conducting a pen test, and they want to forward the pen test findings so that we can secure their environment for them. And there are times when we have noticed that something is going wrong on the customer's end, and we take efforts to fix that for customer, or like we coordinate with them to fix it. So it's a process where we interact with the customer and the researcher, and both of them benefit out of it.

And I'll add one more piece to that, because he made me think. Once we get enough of a specific type of report, we do have this idea of pattern analysis and root cause analysis. So a lot A lot of times, the BRP is focused on the specific mitigation of that specific issue that was reported. But we do have a larger pattern analysis that we're looking for so that we make more architectural changes to the product. So I do think even, like, we take those findings and kind of improve on them. So I do think customers benefit in a multitude of ways.

Yeah, I want to give an example for that. If, for what Kote was saying, we were receiving bucket squatting attacks where customer buckets were taken, then like internally we had a discussion where we want to secure it for the customer. Now we are taking actions to like prevent bucket squatting on whole so that even the customer is safe.

That makes a ton of sense. So I hate to do it because this has been such an interesting, wide-ranging conversation and it's always fun to talk about vulnerabilities, but I have to ask our closing questions. So, One, do you have a recommended tip for people who want to get started looking for vulnerabilities in Google? And 2, do you have recommended reading for people?

So for getting started, there's a lot of— Kodo, who did the Google VRP, has a lot of articles out there, and I'm going to quote him. He said, do with what you know. There's all sorts of vulnerabilities within our products that we are unaware of at this point. So if you work day in, day out with a product, You're going to be able to better identify those. So don't try and learn something brand new. Start with what you know. I personally have no readings that I think would start, but what about you, Adarsh?

So I would say the best place to start would be reading the blog on Bug Hunters. So we do publish CBR findings over there too, which is great reading material because those are internal security engineers who do research on cloud products, and what comes out of their finding is then published into these blogs. So, that's great material to read.

That makes a ton of sense. Darsh, Michael, can't thank you two enough for joining us today.

It's a pleasure. Thank you for having us.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, Or wherever else you get your podcasts. Also, you can find us at our website cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikin and TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.