Security Teams Are Gaslighting Hackers (Vulnerability Disclosure) - Hackers On The Rocks Podcast

Feeling thirsty for knowledge? Hackers on the Rocks is the podcast that mixes refreshing conversation with potent cybersecurity insights. Serving up these insights with a twist, we pair a different software vulnerability with a refreshing cocktail recipe each episode. All right. Hello world and greetings all around. Today we are going to talk about a topic very dear to my heart, uh, and that is vulnerability disclosure. We have two amazing guests. Each bringing some solid perspective. Uh, we're very excited to get started. Caleb Lurch, Casey Ellis, gentlemen, welcome to Hackers on the Rocks.

Thank you, really excited to be here.

Thanks for having us.

Yeah, so Caleb, as is our custom here, uh, the guest picks our drink. So what do we have today?

Uh, I'm— we're drinking some spiced rum. Um, it's my personal favorite alcohol.

I don't drink that much, but, and if I, if I remember correctly from, from our, our, our pre-call, I think, uh, you had, you had made the, I actually took this down in my notes, like you, you really like white rum, but not the cheap white rum. And so we had to go with something a little bit different for today's show.

Uh, fancy.

Casey, you have a, you have a glass as well, right? Is that, is that correct statement?

Yeah, so I'm actually in Sydney, Australia right now and it's 8:00 AM, so I'm drinking Coke Zero. Which is not bad. Doc-ish. We're good to go.

Fair enough. Well, cheers, gentlemen.

Cheers.

All right. Uh, Caleb, while we're drinking, maybe, um, can you tell us a little bit about yourself and how you got in the research community?

Yeah. So I'm currently working as an application security specialist at Digital Boundary Group in Canada. I graduated from school just about a year ago. And since then, I've kind of been dipping my toes into offensive security wherever I can. I've done some bug bounty. I've done some security security research, getting into penetration testing a bit. And, um, I've really enjoyed it so far. So it's been, um, one experience has naturally led to the next experience, led to the next experience as I just kind of continue exploring the industry.

Dig it. And Casey, let's bring you in here too, man.

While, while Caleb continues to jerk, uh, watch yourself how old you are and, uh, Well, and as the, you know, the caffeine kicks in down here as well. Um, yeah. So, uh, yeah, I, um, was the guy who scribbled on the napkin and created the crowdsourced security as a service model back in the day. Um, so founded Bugcrowd. Um, and, you know, like the whole idea behind that is that there's all of these different folk out there like Caleb that have the ability to discover things and, and, you know, bring them to bear for defenders to be able to actually protect themselves and outsmart the adversary. Um, but in the process of doing that, it turned into a policy conversation pretty quickly, like the whole idea of, you know, hackers being kind of automatically assumed to be bad, um, no matter what the intent was. So I also started a thing called Discloso, which is basically, you know, trying to, um, normalize receiving the input of folks that find broken stuff, um, and, and seeing the laws change as a result of that. So like, we've had some success over the years. It's been like a long journey. Like the, um, the CFAA saw some reform out of the DOJ in 2022. Um, and you know, other parts of the world are catching on with that whole thing. But yeah, that's, that's kind of what I've been up to.

Yeah, rock on. Well, we're definitely gonna talk about all of this in a moment. Uh, for now, let's just take a quick 20-second pause and thank our show sponsor. You found a zero-day. Now what? DesiredEffect.io is your shortcut to meaningful monetization. We connect your research directly to the buyers that value your discovery, allowing you to stay focused on what you do best, finding the next one. Learn more at DesiredEffect.io. All right. Welcome back. Our next segment is called The Distillery. Uh, as said at the top of the show, today's topic is vulnerability disclosure. So I want to tee this up for our audience. Caleb, you found something of concern and your first reaction was to directly reach out to the impacted vendor. Can you start by maybe sharing a little bit about what you found?

Yeah, I found a security solution bypass. For a large enterprise security company that I had some personal connections with. I met them at some events that we go to and things like that. So I reached out to them, let them know what the issue was. And, you know, we, we were able to get a few conversations, um, through, I guess, kind of person to person. Um, we had some connections there. And unfortunately that didn't, it was never brought further to the next level. It was run up the internal chains, I think, without me. And then, you know, my connections response was essentially, you know, it's fine, don't worry about it.

Yeah. And so I want to kind of pull on that a little bit further, right? Because I think traditionally, you know, a common story, at least that I hear, is, you know, researcher finds a thing and then goes to some website to figure out what is the appropriate next step. And in this case, you were actually a client of this customer and you had access to both salespeople and, you know, customer support people. And, uh, so your pathway might not have been their preferred pathway, but regardless, it sounds like it was just a challenge to work with them. Do you have maybe Some examples for the audience of like a specific roadblock that you encountered?

Yeah, this company had a bug bounty program. Um, what I found wasn't in scope. They had a bug bounty program set up for their online web services, not for their actual product that they were running. And the bug bounty program, like the terms and conditions I felt were very adversarial. It was a matter of if you. You know, reach out to the bug bounty program email, then anything you tell us is ours. We reserve the right not to fix it. You agree to sign any terms and conditions we send your way. Um, you won't ever disclose, which is fine. I am, sometimes I could see that being okay. I wasn't comfortable with it. I didn't want to sign myself up to legally agree to something that I couldn't read at that moment was the main thing. So I, um, I talked to some people who were, you know, dealing with a product every day. They were sales, they were support. And, um, I, I just essentially gave them the, um, the vulnerability, the, uh, how to exploit it, you know, for free, just kind of across the table, uh, at dinner. I gave it to someone who didn't have the technical understanding. Um, the— I, I guess you could describe it as an oversight. Um, it was an enterprise security solution that was not properly locking down the Windows operating system, so there was some integration functionality that they didn't realize had an impact to their, um, their product. And so they, the person I disclosed this directly to, you know, went and found on the internal team, a very senior person and told them what I had said, and then was given a script to come back and tell me about why it wasn't actually vulnerable. And the, um, the flaw that I had given them the exploit for was not exploitable.

And so maybe, maybe one final question before we bring Casey in. Looking back on all those experiences, right? The front door didn't look good to you. You kind of reached through some personal connections. They did their own probably non-standard process internal and then came back and said, no, no, there's nothing to see here. Like looking back on all that, like what, what would a win have been for you? It sounds like you, you're not in this specific one for, you know, fortune or fame and glory. You said you've kind of given them all of the, the, the write-ups and descriptors needed. What, and you're still not happy, right? So like what, what would a win have looked like for you?

The, the lowest bar I think for the win would have been to quietly fix it and just the, you know, communication with me being bad is, you know, not ideal. I'd like to, you know, best case scenario, yeah, we've made our product better. This is the guy who helped us. Here's some credit for that. Like I can't, you know, put on my resume that I helped them make their product, but maybe I, you know, have contributed in some way that, um, they would be willing to acknowledge.

You found an unintended feature, so there is that.

Yeah.

Um, well, so Casey, so you're, you're hearing Caleb's story and, and Caleb being like, you know, one vulnerability researcher in a sea of many, how, and you've been in the community for a really long time, like how, how common is Caleb's story and Like, what other perspectives do you gather out of this?

I mean, it's super common. Like, I think, you know, there is like one of the downsides to the growth of, like, to me, like vulnerability disclosure, you know, people finding bugs and trying to get them to people that can fix them and then disclose them. Like that predates the whole bug bounty movement by, you know, probably two decades or so, if not more. And one of the things that happened when, you know, we started to introduce this concept of paying for bugs is that like, that's a, you know, fee-for-service type thing, you know, that introduces the idea of NDAs and it being kind of a contracted piece of work. And it's, you know, I think both things can exist at the same time, but they often get conflated with each other. So like if someone's out there on the internet, they're doing their own research like Caleb did. And, and discover an issue, um, it's ultimately their prerogative around what they do with that bug. Like, they can exploit it, they can sell it off to someone who's going to weaponize it, they can try to get to the vendor. There's all these different options that are available. And I think, um, you know, in Caleb's case, in terms of what I know about it, um, you know, the, uh, vulnerable party involved, um, was doing the equivalent of walking outside when there was a storm coming towards their house and saying, don't hit me with lightning. Do you know what I mean? It's like, that's not how this works. It's just a thing that can happen and you need to prepare for it. You can't assume that you can control how that information flows off the back of that. So like, you know, for organizations that are trying to build software, trying to receive security feedback that want to make their products secure, like being proactive around actually receiving input from folk like Caleb, I think is the most important thing. And then setting expectations around how it's going to work. Like, who do you talk to? You know, are you going to get stuck with the marketing team or the legal team?

And is that essentially what the disclose.io project is essentially trying to do?

Yeah, yeah, basically, like trying to get organizations to recognize that like to err is human, right? Like humans write software, you know, we're in an age now where AI is accelerating all of that, but at the end of the day, humans still push the buttons that, you know, get the software out to prod and get it in the eyes of, you know, folks that can use it, but also folks that can potentially exploit it. That's never going to be perfect. So, you know, this idea of being able to receive feedback that you might not necessarily want as an organization, but you need to have and you need to be able to deal with in order to make your users safer. I think that's a really important thing. Yeah. And that's what we've been trying to do with Sqoza, like the whole idea of like, you know, companies that kind of recognize the fact that that's how this all works and, you know, are proactive about that. Like they're the ones that are actually kind of more mature from a security standpoint, and they're the ones that can be trusted more ultimately.

Yeah, it's funny. What you just said actually mirrors something that you've said a few times. I've heard you say, like, I love this that you've said this, but, you know, we've, your words, we've come a long way in terms of recognizing that, you know, hacker input, hacker feedback is in fact a strategic asset.

Yeah.

And trying to mature businesses to values at where to, to, uh, I think both of your points, Caleb and Casey, you've kind of both said that, you know, there, there are good ones and bad ones and vendors historically have had a knee-jerk reaction of, you know, cease and desist and nasty grams and legal reprisal. And how do we get past that? 'Cause it's, it's 2026. Like why, why are not, why aren't all companies the good ones? Like why are we still dealing with issues where vendors just don't want to admit that there was an issue or thank anyone for introducing an improvement request.

I mean, I'd love to hear Caleb's kind of take on this as someone on the breaking end of things. Like I can speak to it as a broker. Like honestly, like to me what it comes down to is it's never nice to have your baby called ugly. Do you know what I mean? Like having someone show up and say like, Yeah, you've got a problem. I can break into your stuff. Like you've, you've, you know, ultimately failed in some way. Um, but what I'm trying to do is to point out that failure so that you can correct it and improve going forward. Like that's a really valuable thing. Um, not every organization is ready to hear that still in 2026. Do you know what I mean? Like it's, it's one of those things where this is a. Inherently adversarial conversation to start in the first place. And I think a lot of organizations kind of react based on the nature of the subject matter. It's like, yep, we could break into your stuff. Like you've screwed up, it's broken, you should fix it. That's just how it is though. Like, you know, the idea of that being possible and that being a part of like what software development actually looks like and, you know, folks that can break into that stuff. Coming along and trying to help out. Like, that's the reality of the internet. And I think the more that the behavior of the internet reflecting that reality looks like, the better off we'll be.

So, Kaleb, I, I know that you have— like, we've started the show talking about one specific thing where the vendor experience, your perspective as ever, experiences has not been great. But I know you've also found some other things and have worked with vendors that have provided a positive experience. Can you maybe share like what encapsulates or defines a challenging relationship versus a successful one for you?

I think, I don't know, I'm coming at this relatively new to the industry. I'm definitely approaching most disclosures as the, hey, I'm pretty sure I found something. Hopefully this is something that the vendor also cares about. Um, that way it was a good use of my time. That way my time has some positive impact in the world. And then, you know, I would love to see this fixed and the more interaction or communication that I can get along this process, then I feel like the better I will be next time. So if this is, you know, your peak care as the vendor, if this is, yeah, I'm gonna fix it, but if you wanna spend more time on my product, I'd really love it if you went over there a little bit. I mean, I was. Exchanging some emails with the Mozilla security team recently, trying to decide whether or not a vulnerability qualified for a bug bounty. And they essentially shared their whole internal discussion and playing both sides in what their decision was, right? They're going to fix the issue. It didn't qualify for bounty. This is why. This was our entire internal discussion. Um, and that extra communication really, I mean, in my opinion, it also leads me to want to keep working with their program. Um, but it helps me, uh, align myself as a new researcher to put my time and work into where I can have a positive impact, where the effort will, you know, trigger some kind of change. And I don't want to really spend all my time researching a vulnerability that nobody wants to see. Um, I just personally, I'm not saying nobody else exists who does want to do that. Um, but if I spend my time researching a vulnerability and I can't say at the other end that I've had a positive impact, I consider it very much to be a waste of my time.

Yeah. What's the point? That's at that stage, right?

Yeah. Yeah. And Casey, is that, does Caleb's story mirror, I know that you've come at it from a, from a broker standpoint with, uh, you know, with BugCrab and now Disclose, but Do you, like, to you, what does the ideal researcher-vendor relationship look like?

I mean, like, I think Caleb nailed it. Like, you know, the whole idea of, you know, as a researcher who's found something that you think should be fixed, you don't want to feel like you're shouting into a black hole. Like, you know, to start with, you want to be able to understand who you should connect with in the first place. Um, you know, the idea of like reaching out to, to different folk. you know, in whatever team within the organization to try to get a thing done. Like that's really common and it shouldn't be the way that it works. Like you should ultimately be talking to product people that have, you know, authority over the engineering team to, to get a thing fixed because these are, you know, undocumented features that we're talking about here. Right. Um, so how do you put those in the pipeline? How do you queue them up to get fixed? How do you make the user of that software safer? And then, you know, for the vendor, like, how do you make it obvious for folk like Caleb and folk that are doing security research? Like, if you find something like that, who do you talk to? Like, who do you get in touch with and how do you actually initiate that conversation? I think beyond that, you know, it really comes down to like, what can people expect when they do that? You know, are they going to get their door kicked in? Are you going to like, lawyer up and, you know, call the police because this can look a little bit like what the bad guys do. In the process, like, if you've got an established, you know, invitation for adversarial feedback from the outside, what's the expected experience of people that actually are, you know, really smart and like actually discovering issues that you've maybe missed and are trying to help. That's not a common, you know, like, again, we've made a ton of progress around this whole thing over the past, you know, 12 or 13 years, but it's still, you know, far from done at this point in time. So like making that stuff obvious and then, you know, making folks that are doing security research feel safe as they're doing that, but also like making them you know, understand what the process will actually look like and, you know, giving some comfort around the fact that it'll be fixed. Like I said before, like the growth of bug bounty and the idea of like getting paid for a bug, I think has conflated with this general idea of vulnerability research being a part of how the internet gets safer. right? Like, you know, the, the folks that discover bugs in like Microsoft products, for example, they're not always selling those bugs in that process. It gets, you know, through to the MSRC. The MSRC like deals with it. They've got an established kind of process to, to work that into the, the product pipeline and, you know, get it to engineering so they can fix things and do regression testing and all the stuff that's involved in fixing a bug. That has nothing to do necessarily with whether or not that researcher gets paid. Like ideally they should, I think, because every bug's a startup. But, you know, if it's not, then there still needs to be a way to do that and to have, you know, folk like Caleb not feel like they're shouting into a black hole.

Maybe building on that answer, Casey, you know, and also building on your 13 years of experience here. When we heard earlier about Caleb's struggle, um, where he's bringing his recommendations directly into the inside of this one vendor and their response is, oh yeah, we, we looked at it. Definitely not a problem. And Caleb can come back and be like, well, it's, it's still to this day not fixed. I guess maybe there's a two-part question, but like, how does a researcher determine if this is just like a simple misunderstanding? Like he said, there's a non-technical receiver of the information or like or when is that vendor actually acting in bad faith? And I guess, does it even matter?

Well, I think the first thing there is to not assume bad faith, like never, you know, attribute to malice what can be adequately explained by inexperience or stupidity or naivety, right? Like Hamlin's Razor, that's a real thing. So, you know, to me, um, probably starting off there, you know, the reality for the security researchers, they don't understand all of the things that going on inside the business. Like, what are the equities involved in trying to actually deal with this stuff? Like, what are the costs involved in actually fixing a bug? You know, the challenge with being a hacker is that, you know, we're very good at getting to a binary conclusion around like, can we get shell or not, right? But that doesn't accommodate all of the other things that are going on. Inside the target organization that we're exploiting at that point in time. So, you know, just starting there is probably a good thing. Like, I think empathy is kind of a key to a lot of this stuff. Um, but, you know, very practically, um, I think vendors should ultimately, if they've got a VDP, if they've got like a recipient, you know, vulnerability disclosure policy with Safe Harbor and, you know, kind of setting expectations for researchers and all those different things. They should also have a CVD timeline. So like, if we don't fix this, if we don't touch this within a certain period of time, then you're free to publish it. And if it's BS, then, you know, the public's going to call you out on that. Your hacker peers will say that's a silly bug and you shouldn't have made such a big deal about this, or it'll make them accountable to actually, you know, produce a fix. Yeah, I think for researchers actually saying like, this is what I'm going to do. In terms of actually publishing this thing proactively, if that doesn't already exist, if there's not, you know, a policy that's there that says that you can publish this and we're going to, you know, lawyer up on you or do whatever else at that point in time, like letting them know that you're actually going to publish within a certain period of time. And then allowing the conversation that'll follow from that because you're putting them on the hook, right? Like if it's a pacemaker or a satellite or a, you know, something that takes a long time to fix, it's probably going to be longer than 90 days. But if it's a SaaS platform or, you know, some sort of install software that has like an auto-update routine, 90 days is probably enough. You know, making that clear when you send the bug across, I think is a good thing.

And just real quickly, just as kind of a thing that we do on our show, since you're bringing up terms, I just want to make sure that the audience is defined on that. Can you explain the concept of safe harbor and bilateral safe harbor? Yeah.

So, um, well, I mean, it's very dynamic. So no is the short answer to that. Um, but you've, you've got to, you've got to be, I think, as a security researcher trying to help folk out, prepared for whatever you might get from like a blowback standpoint. And that's just the nature of the game. Like, we're trying to reduce that and make it more stable and make it more kind of normalized for, you know, security research to be kind of acknowledged as a part of the internet's immune system. And kind of like you said at the start of the show, like hackers are a strategic asset when it comes to the resilience and the antifragility of software. So the more organizations can like proactively recognize that and engage that, I think the better. Um, but as a researcher, you've got to assume that they're not there yet. Um, and that's just kind of the world we live in right now. And, and, you know, the better world that we're trying to get towards. So, you know, being in a position to say, hey, I found this thing. Um, I'm going to publish it within a certain period of time. Do you want to talk about that timeline? Like, how long is it going to take you to actually fix this? And, and, you know, determine whether what I'm telling you as a security researcher is BS or not. Um, you know, like, that's going to be a conversation that can get kind of annoying, I think, for the first time. Um, you know, if you've got a, a vendor that hasn't actually been subject to a lot of this in the past. Um, but ultimately what we're trying to do with Disclose.IO is to get as many vendors as possible to the point where they're acknowledging the fact that this is how the internet works and like we should actually be proactive about this and kind of roll out the red carpet for folk like Caleb.

Yeah, I appreciate that. And Caleb, I definitely, I echo Casey's sentiment and I actually, both of you, I love that both of you are not letting what we'll call a bad vendor without naming vendors specifically, just recognizing that they, there's a long way for the relationship to be repaired between researchers and vendors. We've come a long way, but there's still a long way to go. Uh, but despite that, there hasn't been a chilling effect, right? Some of this, uh, uh, you know, again, uh, in Caleb, in your words, right? Like, uh, oh, there's not a problem. There's nothing, there's nothing serious working as designed. Go away. And even if you went through the regular channels, like, NDA and, but that hasn't, that hasn't quelled your, your inspiration and your appetite for finding other things in other pieces of software or other systems. And you're not going underground and, and, you know, becoming one of these allegedly bad faith actors that the vendors are always whining about. So I just, I don't know, I don't know that I have a question here, but just, it, it sounds, I, I'm, I'm heartened by the perspectives that both of you have, have, uh, kind of, uh, brought to the audience today. Hmm. All right.

Yeah, I think, um, as the researcher, like the way that I self-consider this, and maybe this will change, maybe this is a point in time opinion, but I essentially, my self-talk goes, my responsibility ends with the communication, right? I can do a good work when I find an issue, I can You know, do my best, maybe some good moments, maybe some hard moments when I'm communicating the issue. And at that point in time, I'm more or less ready to just move on to the next issue to look for. Um, and I'd love for everything to get fixed. Uh, if something's not getting fixed, I'm not going to spend any more time in that particular area. Um, product doesn't belong to me. I could tell you all what I would do with lots of products if they belonged to me and I was the number one, um, decision maker. But I also think, you know, to that end, I don't feel like it's my responsibility to prevent someone else from finding this. I don't feel like it's my responsibility to not, you know, let my friends know who are using this product so that they can defend in some other way.

Okay.

Um, this is just, you know, I find the issue, I disclose the issue, and that is where I personally feel like the responsibility kind of ends. Kind of goes back to the world of the next hacker or the vendor or, you know, the consumer if they come across this themselves as well.

Yeah, it's funny. I got roped into a, you know, a debate at a conference one time around, you know, full disclosure, like, is it good or bad? And, you know, my entire point, like, I was actually on the for side, which is not what I'd normally be associated with, right? And I won. And, you know, one of the reasons I think that happened was, you know, full disclosure is like death and taxes. Like, you can dislike it all you want, but ultimately it's a failure in the system and a failure in that conversation. Like, if an organization is not prepared to recognize the fact that they're going to get feedback that they might not like about their product from the outside world, and they can't deal with that. And if ultimately they try to silence that, which they often do, you know, the last kind of resort for the researcher that's in your position, Caleb, where you're trying to, you know, make sure that, you know, your peers that are using the same product are safe is to drop it on the internet so they can defend themselves. Like, that's not a, you know, like, to me, that's a gravity issue. It's not so much like a CVD thing or a VDP thing or a bug bounty or any of that other stuff. It's the fact that like you've tried to tell them how to make this stuff safer. They've not listened. And, you know, you're left without any other choice at that point in time, but to put leverage on them to actually fix the thing. And in the process, you know, share with your peers. Where they're unsafe. Um, that's kind of, to me, the starting point of all this, if that makes sense. Like, that's, that's kind of what we're dealing with as, like, you know, the, like, nature's way of solving this problem. And it's not perfect because, you know, at the end of the day, like, when you do that, every other bad guy gets a hold of that data. Um, they're in a position to actually exploit it. You know, you've got actual bad folk that can use the things that have been found to do bad stuff, that's not a good outcome. But, you know, the way to avoid that is to be able to receive that input from the outside world and actually do something with it.

Completely agree. Uh, last call, uh, Caleb, where can people go to find out more about you, your work, anything you mentioned today?

Uh, probably LinkedIn is the best thing to find me. I'm Caleb Lurch on LinkedIn. There's a link to a blog on there. Um, I go through phases of ignoring all invites and then accepting all invites. When I— people try to reach out to me on LinkedIn, but if you shoot me a message there, I'll get around to responding.

That's funny. I'm the same way. Casey, same question. Where can people go to find out more about you, your work, anything you mentioned today?

Yeah, as a side note, Caleb, if you delegate that to Claude, it makes life a whole lot easier. Let the AI handle it. It's fine. Um, yeah, so I'm at cje.io. Bugcrowd is at bugcrowd.com. Um, Disclose.io is at disclose.io. Um, you know, for organizations that are looking to, you know, basically implement a policy and a framework to be able to receive input from the outside world and make it safe for them, like what you're not saying is, hey, internet, come hack me, do whatever you want. What you're trying to do is to say like, if you stay within, you know, the reasonable definitions of good faith, secure your search, and then you find something in the process of doing that, this is how you tell us that, and this is what you can expect when you do. That's really what we're trying to do there. So yeah, that's where to find me.

Gentlemen, I appreciate it. Thank you guys, both of you so much. Really appreciate today's conversation. Caleb, it's me, your buddy. Cheers, my friend.

Thanks. Cheers.