EP253 The Craft of Cloud Bug Hunting: Writing Winning Reports and Secrets from a VRP Champion

Security running.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Tim Peacock, the PM for a bunch of stuff in Google SecOps, and of course, Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You follow the show, argue with us and the rest of our Cloud Security Podcast listeners on the Google Cloud Security Community page. Anton, we are— well, this is just like a delightful episode. We're talking to like— I don't even know what to say about this one. What are we doing today?

This was really fun, but it's also, to me, this episode combines the fun and the absolute utility, the value for something people can actually use. I'm going to sound like a low-end, cheap marketing person, but they can literally make money with that advice.

Listeners, Anton as a utilitarian confirmed. Finally, we all know what his philosophical leanings are. But no, you're right. You could literally listen to this episode and then go make money by becoming a Correct.

And while I feel also their advice is also something that a seasoned bug bounty hunter can probably use in some areas, but it's also really, really useful for people starting up. And I feel like this is— we are maybe now an education business, Tim, because we are telling people how they can start, not only telling people how something is important or how to migrate from their 15th SIEM to their 16th SIEM or whatever.

Oh, God. Sad that that's the business we've gotten ourselves into. But also, listeners, I think there's some really There's some really interesting organizational psychology that's relevant not just for Googlers, but for anybody running or working in a large software engineering company. There's a delightful story about sliding into somebody's DMs. I mean, this episode has it all. And so maybe without any more spoilers, let's turn things over to both of today's guests. Today we are joined by 2 guests, so get ready for a doubleheader. We've got Sivanesh Ashok. And Sriram KL, they're both full-time bug bounty hunters. They don't, they don't have like real jobs. They are full-time bug bounty hunters, which is amazing. Yeah, they do. I love this. So we were talking to the Cloud Vulnerability Rewards Program team and they said that you two write excellent reports. Could you tell our listeners, A, what makes your reports so good? And B, what advice you would have for people who are submitting reports to Google? to also write excellent reports?

First of all, I'm glad to hear that feedback from the team. We do put in quite a bit of effort into the reports. So, you know, when the team appreciates it, it really, we really feel good about that. So for us, the focus on writing the report is like when we are, you know, we found a bug and we have to report it now. The focus is not about giving all the technical details of why this bug exists, et cetera, et cetera. We do include that information, but our primary focus is actually focused on the triager, the person who is reproducing our reports. We want to make sure that we provide all the information and all the steps that we can to make the person who is triaging our report, their life as easy as possible. That's primarily the point that I would add. After that, once the person's like, okay, I've got the bug working, now why does this bug exist? Then we can give all the details about nitty-gritty details about the bug itself. So yeah, that's the concise gist of it. For the listeners, like for anyone who is doing bug bounty, that's the best advice that I can give them. Like, you know, folk really give the triager some attention, like instead of just talking to the product team and how to fix the bug and et cetera, et cetera, just let them first reproduce it and then yeah, move on.

Yeah.

So I want to go kind of a little off script on this. In my entire security career, I was pretty far from bug hunting. But I've dealt a little bit with vendors responding to bug reports kind of in bad faith. So this is like a very Anton thinking about stuff question. But imagine you are a researcher and you're reaching to a vendor in good faith to report a bug.

Yeah.

And the vendor has 2 choices. They may say, we honestly don't believe it's a bug. Or they secretly believe it's a bug, but they say, nah, it's not a bug. How do you react to this? Like, how do you know? Are they? Do they mean it? Are they quote unquote the good guys, or are they trying to hide something?

This usually doesn't happen with Google because Google's.

Oh yeah, I did not plant the question so that you can say that.

We believe you. Okay, it doesn't happen very often. At least you you go with good programs, and I mean this had happened to a lot of people, and people cry about it a lot, and. It's pretty common, but it doesn't happen with a lot of reputable programs. So that's one reason we, if you don't know, but we always stick with Google VRP. We don't go away from Google's other programs, like any, like some obscure program, because for the same reason, they may not give you enough attention or they may not treat you the same way, like really reputable bug bounty programs do. So to avoid those situations, we go with good programs. For now, we only run on Google programs.

Really?

Yeah, we are full-time bug bounty hunters on Google programs.

Just Google?

Just Google.

That's amazing.

And you can obviously take a very positive, beautiful take, and you can have a cynical take that our bugs are enough to make a living.

Yeah, it's a question that, you know, that's a topic that comes up very often, actually. When we tell people that we are doing full-time bug bounty, it comes up. They ask, like, is it enough? And I would say that we both got lucky with the timing when we started because I started off bug bounty when I was just right out of college. I did not have any immediate pressure to like, oh, I have to make money, I have to get a job right now, like right away. I wasn't under that pressure. And Sriram was also a student at the time, so we had some time to build like an experience in bug bounty.

Mm-hmm.

Which let us coast off now. Like right now we can, of course we can sustain. We've been doing it for like 4 years now. So, you know, we can sustain it now, but if we were in a full-time job and then we have to like quit the job and go into bug bounty, that would be hard. I would say that that was, that would be hard.

Yeah.

So let me ask you another, probably it's not a very Google-serving, but it's kind of more of a you-serving question. So you recently won this Most Valuable Hacker award at the event in June and It's interesting to me that what seems like a technical skill really isn't just a straight raw technical power, right? Like, you can train others, you can teach people where to go, you can teach them to use tools, but then the one guy, well, you would be best and another guy would be just okay. So, what is the magic? What makes you successful in finding issues? You can be as self-serving as you want. You can say that you have secret magic inherited for generations in finding bugs. I believe it.

So I would say one primary power that we have, that again we have, is that we collaborate. We do full-time collaborations. So I don't think many bug hunters who are doing this, they rarely collaborate. Even if they do, they don't full-time collaborate. So one, so, you know, again, it's a secret sauce. Every bug hunter has the secret sauce and they don't share it because it's a money-making machine, right?

Yeah.

But even if at some point they're going to collaborate with some hunters for some reason, they don't fully share their information or the knowledge that they have gained through years. They will share some part of it, but they might have the secret trick under their sleeve. But, uh, we don't have to do that. So that is a big thing, I would say. So for clarity, we split all the rewards by 50/50.

Mm-hmm.

So it doesn't matter who finds a bug. We share it 50/50. So we see it like a full-time job. We both work on it. We take 50% from any bug. So I don't have to hide anything from him. So even if I'm hiding something from him, that doesn't make me money. So it puts an incentive to share ideas and brainstorm and basically share any resources with the other person. So I think that is a big secret sauce that we have. I mean, it's not secret anymore, but— I would say that.

I would say that.

That's it.

Oddly enough, that makes sense because I've seen examples and I've met people who basically very much manifest 1 plus 1 equals to 4 or 1 plus 1 equals to, you know, some other larger number but not 2. And I think that to me makes logical sense.

Yeah, yeah. To add to that, I would say it actually, it adds some kind of stability as well because sometimes one of us may not be able to work. Maybe we are busy with something else in life. So, but still, like, it takes off a little bit of the volatility in bug bounty itself. So the other person is working. If he's not working, I'm working. I'm not working, he's working. And it kind of like helps us coast along. Yeah.

Volatility. That's an interesting topic. Let's talk about that since you're doing this as a full-time job. This is also not on the script, listeners, but Tell me about the volatility. Like, to me, this would feel so anxiety-inducing. Like, I don't know what I'm going to find the next thing. So how do you, aside from the teamwork aspect, which obviously makes a lot of sense, what else do you do to deal with the volatility? And is it maybe not as volatile actually as I'm imagining?

So it's very volatile, more than anything.

It's very volatile.

Yes. But the problem, but an advantage for us is, I mean, we have been hunting on Google for so long that we know so many products and we know so many behaviors and bugs. That Google tends to recreate.

Okay.

So it's like where we know where to look for because we have gained this experience for a very long time. So for us, it's like we usually find more bugs than we report because it's like we find bugs, we do report it, but most of the time we get lazy. So we just note it down somewhere and we just sleep on it. But it's not usually that bad because you might think it's it's a security issue, it must be fixed immediately. But usually bugs we find are bugs that mostly only us, only we do find that because it's, it's very— I would say our bugs are very unique that we rarely get duplicates.

Hmm.

So although it's a security issue, I think many bugs that we at least hold are the ones that others don't usually find. So a threat actor actually using it is also a very— the chances are very low. So We just write it down. So when we don't find any bugs, we just go look at the start and we say, okay, I have like 5 bugs to report. So we just report it and call it a day.

This actually, I find very relatable because Anton and I, we get inbound interesting podcast ideas all the time. People write in, they're like, hey, we want to come on the show and talk about post-quantum crypto, or hey, you should talk to these 2 guys that find all these bugs on Google. And other times nobody writes to us for a couple of weeks and we've got this big spreadsheet Full of episode ideas. And then Anton and I use our one-on-one to like sit there and go, uh, which of these should we take out of the Siberian freezer today? You know, that's, that's actually how we manage our show too. So this makes a lot of sense to me.

It's always a spreadsheet, right?

It's always a spreadsheet. Spreadsheets are the duct tape of product development. That's my—

Oh yeah, yeah. We have like, I would say maybe like 5 or maybe even 10 spreadsheets of different stuff, just random stuff. And sometimes it's just, you won't even remember it's there. And then like, yeah, it just keeps on giving. So yeah.

So I'm very curious about this, this thing you're saying about like, you know where to look for Google bugs. Have you reached like psychological conclusions about Google as an organization based on the kinds of bugs you tend to find? Like, do you understand us better than we understand ourselves? In other words, like, what insights about Google can you share?

Well, I don't know if we understand Google more than you guys, but we—

No, no, no, wait, wait, wait.

I believe that you do, and here's why. Because we are actually very many people, and some of us don't talk to each other. So you don't even need to preface this because it is very possible that you understand some of our bugs better than us because us is too many people.

Yeah, there's way too many of us.

Makes sense, yeah. Yeah. So we kind of like expect Google to make certain mistakes, like the product teams. So when they implement a certain feature, which we have like exploited in so many different products, and we are like 90% sure that the same bug is going to be there because just like you said, the product teams don't really like interact with each other, each other. So it's not like they are going to look at, you know, okay. this product had this feature and, you know, they fixed this security issue. So maybe we should also focus on it. No, it's not happening.

No.

For some bug to get like fixed, fixed, some bugs, it actually needs some kind of like a browser change or like some systemic, you know, spec change or stuff like that. So yeah, so we do, you know, know certain behaviors in Google, like not just in Google, it's just general mistakes that people tend to make. It's like, let's say they are implementing OAuth, for example, like it'll be OAuth on paper. It sounds like very simple, but it's actually like developers make a ton of mistakes when they're implementing it. So yeah, mistakes tend to happen when they're like, you know, adding more and more features. And we, that's how, like when we see certain features, we know like, okay, this kind of mistakes can exist there.

Yeah. So let me ask you, I have an interesting kind of a cross-pollinating question. Tim, remember the question when we talked about the guy who built an AI that can look for bugs, right?

Oh, I wondered if you were going to ask this. Yes.

And his AI looks for kind of unknown bugs, but it tends to find mostly authentication bugs. And we were really puzzled as to why that thing mostly looks for, mostly finds auth bugs. But is there, not at Google, it's like in general for his tech, but is there any, like, do you, each of you have a preferred type of bug that you usually find?

Right. So we tried to find bugs that, that comes with integration, I would say, to put it simple. So we try to find mostly in integration because when, let's say there are 2 different products built by 2 different teams, each product has their own threat model. So they might think, okay, this bug wouldn't happen because we implemented something. But then when these 2 products are merged together at some point, which they'd never expected. to happen, their threat model starts to fail. Because these are 2 different minds making 2 different products, right? So they might think, okay, this is safe. And that might be actually safe. Then the other guy thinks, okay, my product is safe as well. But when they come together, that's when the problems come. So we've tried to find products mostly in integrations. So that's our go-to kind of box that we usually look for. So there are a lot of bugs that are— most bugs are mostly from integration, let it be first party or third party. Let's say someone from Gmail is going to create something in cloud and these teams have never been impacted. I would say like they, they don't know what the other team is doing and they might like, let's say create a new feature. Let's take for example Gemini, as we're talking about AI, let's say they create a feature to unlock your phone and it doesn't exist, but let's say that, and they wouldn't care about the Android security posture because, because that's not their job. But I mean, the Android guys wouldn't expect you— an AI to unlock your phone. But this is when the problem comes. It's not the problem with Android, or it's not with Gemini, but when they come together, there will be bugs. I mean, at least.

So this, I love. I absolutely love it because it's been kind of a very common theme that it's the seams between things and the integrations where things get really lost in some horrendous way. And then if somebody reads a report 2 years later, he would assume that developers are complete idiots. But it's not that at all. It's that people— there was little tiny areas where really solid experts in X and solid experts in Y left a gap. And it is the area where none of them are solid experts, and it's kind of like a wobbly area, and that's where the bug lives. That, to me, somehow just feels philosophically right.

I think that is right. And when we talk about why security is such a different domain than other domains. It's— I really think 2 factors make security different than other domains. One is the thing you just described. It's all about the intersections of systems that actively don't want to know about each other. And then 2, it's actually also the presence of people like you who are really clever at finding those gaps and exploiting them. So I want to bring us back to the VRP for a minute. And could you tell us and tell listeners who are interested in the VRP or interested in running their own VRP, maybe, what do you like about Google's VRP? What do you not like about it? What's your experience been there? Because we could do Google psychology all day, but I think we want to talk about VRP just a little bit, at least.

Yeah. So like I mentioned, we've been collaborating for the past 4 years and exclusively Google. We've submitted like half— yeah, thank you. We've submitted like a handful of bugs to other programs outside Google. That's it. For context, we have submitted hundreds of bugs to Google. So, and for, and for him, Sriram, he's been doing bug hunting with Google even longer. I think like 3 years, 2 years, even 2, 3 years prior to that. So we've almost never had any instance where Google was like, you know, we, where we submit a bug and then they disregarded the bug or they didn't treat it right or give enough attention to the bug. It has never happened. So every single bug is treated on its own merit, I would say. So that is probably the best thing about Google. And if we ever have a question about the reward or the classification of severity or whatever it may be, they will get an answer from an actual human being, like, who is on the other end. Whether we agree with it or not, that's a different thing, but we can still, like, talk to them, like, about the bug. We can have a conversation about, like, what we think about the bug and what they think about the bug. So yeah, I would say that is the primary thing that Google's doing really, really well. And also, like, the response time is really good. We always get a response in couple of days, maybe like a week, but that's the norm here. So, yep, you're right.

And another thing I would add to that is recently Google did a 5x increase in their bounties. So that's like, uh, yeah, so that's like a 5x increment for us. So since it's like a full-time job for us, and again, to answer your question, I would say that we, we try to find very bad things about Google. Like, it's at least, I mean, part, I would say, like to give suggestions because the program is almost perfect. As we said, like we have a sheet for everything and we have a sheet for suggestions for Google VRP. So the sheet is almost empty. The only thing that we have is like add more swags, stuff like that, that we try to fill it, but we rarely can do that. So it's almost perfect, I would say.

That's awesome. I— how did you two meet?

I was hunting on Google for like 8 years now. So like on the 3rd to 4th year, I was on Google, uh, we are running on Google VRP. Then I think Sivanesh had a bug that he want to— he doesn't have a full chain, he had a part of a bug and he wanted someone in Google who has been handling on Google VRP to help him out to finish the bug. So we had a common friend and Sivanesh actually reached out to me on Instagram and it's, it's unlike this.

That's amazing.

That is a fun story indeed.

Yeah.

And the rest is history? Did you end up chaining that bug together? Did it end up working?

No, not really. We tried to do that, but we didn't end up finding the bug in the first thing. But we found some other bug that we somehow— we accidentally collaborated into finding another bug. But then later, and then we have been collaborating, and then later, After one year of us collaborating, we actually finished the first bug. So we found a full chain for the first bug, and then we reported, and the rest is we just tried to do the same thing again.

That's awesome. I love this story. What a fun episode.

Yeah.

So, but we are at the same time, we managed to talk for like 20-something minutes, and we have a couple of questions to ask you before we close. And I feel like We should give advice to people who want to be you. By the way, I'm just reminded, somebody told me to also give advice to people who want to be us, Tim and me, but we'll do it on some other show. But what about, like, a single best piece of advice you would give an aspiring cloud bug hunter today? Like, just give, like, not necessarily how to become you, maybe they have their unique secret sauce, but so what's your best piece of advice to aspiring bug hunters?

So the one thing, if I have to give one advice, it would be— I mean, it sounds cliché, but that's the truth. I already kind of alluded to it in the beginning. It's to actually stay consistent. It's very hard because when you start out, you kind of feel like there is no bugs. There is nothing. Like Google, it's like a fort. It's all secure, right? But you have to spend enough time in your own discipline and in your own target enough time to learn and absorb the things that you're looking at. Just a couple of days is not enough. Even for me, for, for us, when we switch to a new target, we will not find a bug for like maybe like, you know, like a week or something like that. We have to give it some time and like, let's just like marinate in the target so that we can understand how the product works, how all the features working, and then slowly, you know, understand the threat model that's an important thing. So yeah, the biggest piece of advice is time. Like, be patient and like put in the time and effort. And no matter what your discipline is, like, we hunt in a specific way, other bug hunters do it in different ways. It's all like, it comes down to you being like consistent day in and day out. It'll work out. It just takes time.

I think this is the, like, a strong vote for like hard work, which is kind of like excellent advice, but also—

Yeah, it sounds very generic, right? Like, I wish I could Yeah, but one thing like I can say to add to this, if someone else who is already into bug bounty, they wanna like, you know, take something away from this, we would say something that helped us a lot was— I actually picked this up from some other bug hunter— it's to collect data. We actually like log all our, you know, Burp history into an Elasticsearch instance and it helps because we, whenever we come up with a new idea of like, okay, we have to test for this, we have like a backlog of years of HTTP requests. So we can just go back and like look at it. So this is for like people who are already into bug bounty. So like collecting data and also we note that we try to note down like every single thing that we are testing so that in the future, like it might not seem like a bug now, but maybe a couple of years later we learn something new. Take a new look at it and then it's a bug then. So yeah, like writing down notes and like collecting stuff about your bug hunting process, it should definitely help.

Awesome.

I love that advice. It's traditional for us to close with asking for a tip, which Anton just did, but could you also recommend reading for our listeners? Something for people to go learn more from, or maybe just a book that you enjoy? You could go either direction with it.

So we usually go for blogs. I'm not really a book person. So we usually go for blogs and videos and things like that. I would say one big advice I would give on resources is, which I usually don't follow, I don't follow, but I know that's the best way, is to do CTFs. So play good CTF, you will learn a lot. You will learn. I was doing a lot of CTFs at some point. I would say the most of the knowledge that I have learned is from CTF than bug bounty. I would say the technical knowledge. The bug bounty is more like the situational knowledge that you get. It works with Google, but if I switch to another program, it might not work there. But the CTF knowledge are very much switchable. You can switch between programs and it's basic. You understand, get a lot of technical knowledge when you work with CTFs. So that is one advice I would give. Play good CTFs. There are a lot of shitty CTFs. Avoid that. Just play the good ones. If you're into them, you know what I mean. And about blogs, there is a community-driven GitHub repository called Awesome VRP Write-ups, Google VRP Write-ups. So we try to add as much as Google VRP write-ups that we find on the internet into the GitHub repo. So you could look the repo up and maybe you would find a blog there. You can also contribute to it if you find a Google VRP report on the internet. try to add to it. And then another thing I would say is I really love this another podcast thing called Critical Thinking Bug Bounty Podcast.

Critical Thinking Bug Bounty Podcast?

Yes, that's by Justin. It's such an amazing resource. They, again, they run a podcast, they have a YouTube channel, they bring up a lot of bug hunters, try to talk to them, try to give their ideas and their general opinions on things, and they try to discuss new vulnerabilities. And I think they recently also have like a lab thing that you can try your hands on. So such a good resource and it's free. So maybe you should use that.

I love it. Sriram, Shivanesh, thank you both so much for joining us today. This was an absolute pleasure. What a joy.

Thank you so much for having us. Thank you. Thank you. We had a blast. Thank you.

Thank you so much.

And now we are at time. Thank you very much for listening and of course for subscribing. Please subscribe so you can get new episodes piping hot And if you love our content, please drop us a review on your platform of choice.

You can find this podcast on YouTube, Apple Podcasts, Spotify, or wherever you get your podcasts. Also, you can find us on our website, cloud.withgoogle.com/cloudsecurity/podcast. You can argue with us on the Google Cloud Security Community site, googlecloudcommunity.com.

You can follow us on X, x.com/cloudsecpodcast. Tweet at us, email us, argue with us. And if we like or hate what we hear, We can invite you to the next episode.

See you on the next episode of the Cloud Security Podcast by Google.