Cybersecurity Vault - A Deep Dive into Cyber Risk with Wade Baker

Welcome to the Cybersecurity Vault. I'm your host, Matthew Rosenquist, cybersecurity strategist and CISO at Eclipse. Today we're going to talk about the challenges and complexities of cybersecurity metrics, and I'm going to be talking with Wade Baker, who is a cybersecurity researcher, professor at Virginia Tech's College of Business, founding partner of Cyentia Institute, an advisory board member for the RSA Conference and the FAIR Institute. And today's podcast is created in part by Eclipse, securing data in transit through any cloud network or device. Welcome, Wade. Thank you for coming. It's always great talking with you.

Absolutely. I've known you for a long time, respected you for a little less than that. I'm just kidding.

It's going to be one of those. No, I appreciate it.

I appreciate it. It's good to chat with you.

And for our listeners, Wade here is a legend in the cybersecurity metrics community. If you have ever heard or read the Verizon Data Breach Investigation Report, the DBIR, you are familiar with his early work. I remember the first release of the DBIR, which was a long, long time ago. We're not gonna talk how old, which it really was a watershed moment in the history of cybersecurity because it was during a time when actuary metrics didn't exist. There was no company willing to share their dirty laundry, you know, data about being hacked or, you know, breaches occurring. So the world, sadly, in that state was left with vague opinions of risk, talks of demons in the closet and shadows behind every corner, and it found itself quite distant from reality. And unfortunately, people were preyed upon. Um, there was a lot of unethical things that happened, and then Like a beacon of light, the DBIR was— it was the first milestone to change all of that. That yes, actuary metrics could actually be gathered, they could be relied upon. It was a real reflection of what was going on, not just some third-party opinion who couldn't even spell security, right? And after years of publishing the DBIR, Wade moved on to co-found Cyentia, a research firm which produces fantastic research and insights, and I will wholeheartedly endorse them. And it was really to continue that pursuit. And I know you've got passion for it, right, of the realistic metrics that enable good business and risk decisions. And to me, that's always been the point of cybersecurity metrics. If that metric can't contribute to a good decision, It's a worthless metric. It may look pretty, it may sound cool, whatever. And again, you know, I've always been a big fan of you and all the work that you've done and what you've contributed to the industry. So it is my absolute pleasure to once again sit down with you.

I appreciate that. I should record that and play it for my kids. No, I appreciate it.

Is there plausible deniability here? Hold on.

But, um, I, you mentioned several things there that the, the, the DBIR is, is one. And, and I, I was thrilled to be a part of that. Um, but it was definitely inspired by, uh, metrics and the desire for better metrics. So I actually, not, not many people know this, but if you were part of a, a little nerd group security metrics list back in the day, which may have been where we met originally. I don't really— okay. Yeah.

Yes.

Um, but at some Metricon, uh, which is where the real nerds went at the beginning of our RSA, uh, years ago, um, I, I sort of had some analysis, um, done and shared it with that group of people of, hey, here's a bunch of stuff from some forensic investigations, and I think it's pretty interesting. What do y'all think? And got super good feedback. And I mean, that group and the mindset of that group and the encouragement of, yes, that's interesting, I think the world needs to see that, really did lead, you know, to the DBIR. It's one of the reasons why we kept doing it and, you know, convinced, um, uh, management at CyberTrust and then than Verizon to, to do it. So, you know, they're, they're, they very much are related. Like, uh, the, the DBIR is a fantastic marketing tool because it has a lot of visibility and that kind of thing, but it didn't start that way. I think I always, I always like to tell people that, like, it started from a legitimate, I want to figure out what's going on in this space and measure some things and, and share that information with people and You know, the fact that it became kind of an industry milestone was fantastic. I'm glad it did, but you know, it didn't start off that way.

Well, I could care less of your motivations. I don't care if you were sprinkled with pixie dust. The reality is you generating that report and getting— and I don't, to this day, I have no idea how you were able to convince organizations to give you that data and trust you to make sure it was anonymized and everything else. But the fact that we actually had real information on the types of threats and risks, attacks, breaches, things of that sort, that was fundamental. And at the time I was working at Intel, I think I was managing— I was just building the security operations center there. And to be able to communicate the risks with real numbers was, it was phenomenal. It really enabled me to continue to drive and make things happen. Good things happen when you've got good data. So again, personally, I don't care your motivation, right? It could have been your kids beating you over the head and you just wanted to get out of the house. Either way, I love what you did. So the end result It wasn't just me. It was every CISO I talked to at the time also said, I use this. This is my Bible. This is where I go to. When is it being released again? I want the next, the day it's released, I'm going to be downloading and reading it.

Yeah, it was very cool to be a part of that and that people allowed it to happen. And I loved, you know, you mentioned the, how we got people to share information. The first year it was just Verizon. Actually, it was, it was legacy CyberTrust investigative response caseload, right? So we combed through that and published. But after a few years, you know, the Secret Service and lots of law enforcement agencies around the world. And then, you know, I think when I left, there were like 70 organizations contributing to it. And out of that entire time, there was one NDA signed for the information sharing that took place in that, which, which is honestly maybe the largest miracle in the, in the whole thing. But there was something about it that for various motivations, people wanted to be a part of it, be seen as being part of it. But, you know, I miss that aspect of it to this day, even though it's 8 years or so removed since I, since I left the project still going strong, by the way. But it's, it's really cool when that many people sort of gel around one, one thing and are willing to contribute to it. And, and there was no money exchanged, uh, you know, and from any direction on that whole thing. So, uh, very fun project. I like to think that it, it made a difference and still does. Um, and, uh, you know, I know, I know, uh, one of the things we wanted to talk about as well. I think the DBIR sort of sparked lots of other, uh, you know, reports and things like that in the industry, some of which are great. I love it. Um, you know, and are helping to, to, to do the things that you mentioned, get more data out there, help people make decisions. You know, others maybe not so much. Uh, you know, so it's increased, uh, I don't know whether it's probably increased the noise-to-signal ratio in a maybe not a good way sometimes.

But it was a silent room before that. So, you know, it did create and spawn, I think, a whole industry where different companies now, we've got all sorts of great reports that come out from companies and it might just be their data. It might be a consortium, it might be for a specific sector, but Again, that communicating and sharing real data, not just fears and made-up things, but sharing the real data has fundamentally improved our industry, bar none, bar none. It absolutely positively has. And it was that DBIR that broke the ice, broke the surface, and now we've got all these other reports that are contributing as well. So hats off to all of them as well. But you had to break through a pretty thick, you know, piece of ice to see the light of day on that one. So, all right, enough stroking your ego, enough of that. Let's talk about our industry now, because again, you have your finger on the pulse of this and different kinds of metrics and different perceptions and challenges. Where are we now? If you had to describe cybersecurity metrics and the state of the industry, what are we doing good? What are we doing bad? What are those challenges we need to overcome? And are there any successes in the back of your mind that are going, hey, we should be celebrating this?

Rewinding back to the DBIR, not trying to retread the ground there, but it's a good starting point. I think the problem, the challenge that we had at that time was a lack of data. You know, we, we needed to make security decisions. We needed to do things. Uh, and we thought we knew what would be good, but we didn't really have any data to help, uh, support those decisions. And I think that's a big reason why the DBIR was, was seen as valuable and became, uh, very well known is because it was one of the early ones to provide data. Um, and I actually don't think we have a lack of data problem anymore. You know, I think the challenge has changed to, of all of this data that exists, what's actually useful? Uh, and in, in many ways is harder. You know, if you're, if you're looking for what do I trust? When there was sort of a few signals that seem pretty trustworthy, I mean, you just went with that because that's— and if it didn't address what you needed, you know, dang, but you did what you could from what was there. And but now, you know, the problem is all of this stuff is out there, seems to cover every possible question I could ever dream up. But you know, what's, what is actually reliable information? What's, what is, what's the information that I kind of need to parse through and take what I can from it and ditch the rest? And then what's just outright trash. And I, and I think that's hard to distinguish for most people because it's all, you know, it's, it's nice and prettied up and presented well in many cases, not always, but a lot of times just from the presentation. Presentation of it and even the language, you can't really tell what's what. So I, I, different challenge. And I think overall maybe, maybe a little bit more difficult to get to actually, you know, which metrics are the ones I need to care about and are, are actually useful for me.

So then how does, I guess, the media play in that? And I'm not just talking professional media, I'm also talking bloggers and, and everything else. In taking data or taking a report and selectively taking pieces of data out of it, potentially out of context, to support their initiative, their ideas, whatever, or to sensationalize it for their benefit. Is that becoming more and more of a problem that you see, or do you think we've kind of hit the peak of that and people are now, okay, I know there's a lot of misinformation, disinformation, I'm not going to believe that, I'm going to do my own research or whatnot? I mean, where do you think the mindset is for the end users around misinformation and disinformation surrounding cybersecurity metrics?

Yeah, that's a tough one.

Yeah, you're not here for the easy questions. No, I invite other people for easy questions.

I think it's complex, which sounds like a, you know, I'm passing. But there's a lot of things going on in, in that. So, so media, for instance, um, over the last 20 years, undoubtedly more media sources, right? Covering a wider range of topics, all of which are pressed to have something new to say, right? To attract the clicks and the audience and all of that kind of thing. Which in general, not in all cases, but in general incentivizes them not to look too terribly deeply into the research to pick out the hard stuff that maybe you would want to know. It's usually they— hey, that's on the surface. That looks good. I'll take that.

And that's a good headline.

I want that for my headline. I'll make that my headline. And the more sensational, the better. And I think even outside of security, that's a problem. So there's nothing new there, but that does set an incentive for the people that are trying to produce content mainly for media buzz and hits to give the simple stuff, right? And they'll stop. So, oh, that's— I got 3 sound bites in this. Let's pack it up. Done. Um, and you know, I, I think that's a shame if that gets too, um, if that trend continues. But I don't— I'm not saying everybody's doing that, but that's definitely an incentive. And then at the same time, you have a lot more companies out there that need to be heard and have a share of voice, so they're incentivized to produce more content And, you know, so this is like, and then the, and then there's so much coming at the media people that they're, something has to elevate it from, from the crowd. So this, this is kind of a snowball effect on, on all of these things. And I think in some cases it, it makes for a challenging environment to actually produce analysis and research that's well thought out, uh, that is reliable, uh, quality, because there's so many sort of disincentives for doing that. I mean, when we do a project at Scientia, I usually tell people it's going to take us 4 months at least to analyze the data and give it its time and really understand it and draw decent conclusions and write up those things. And, you know, I get a lot of people that are, no, no, no, we've got an event 2 weeks from now. You know, we have to have the, we've got to have our DBIR in 2 weeks. And I have a hard time telling them that that's sort of not realistic, you know. Yeah.

Yeah. You can have it good, fast, or cheap. Pick 2.

Right. Right.

So do you think we're getting smarter? Do you think we are potentially moving to a situation where we will start to embrace or codify standards for certain kinds of metrics, either at the government regulatory level or within certain sectors? I mean, if you go to any CISO and say, you know, tell me what your metrics are, your top 10 metrics, You will get 10 different answers, right? Because everybody's different. Nobody standardizes. And depending on their maturity and everything else, it could be way out there, right? They may still be using a map of, you know, pack— bad packets floating over the planet or something, right? Completely meaningless. You can't make any decisions based on that at all, but it looks cool. Okay, great. Uh, do you think we're getting better? Are we gonna— are, are we getting smarter? Do we see less of that? And do we need more standards or even regulation to help kind of keep us moving in the right direction?

So first of all, I have asked CISOs that question, and you're right, you do get different answers, um, for, for each. But, but I, I'm, you know, cautiously optimistic on the 'are we getting smarter' question. In, in general, I think we are. Um, the Conversations on metrics that are taking place now are, are, are much more mature and, and down the road than they, than they were years ago. And I view that as progress. Uh, is that, is it solved? Or have we figured out exactly what the right metrics under various conditions are? You know, I don't think so. And I think to a large extent, that's probably because at least some of it subjective and context-specific, and I think that's okay. But yes, I think we're, I think we're smarter, and people are asking better questions, demanding, you know, not just how many open vulns do I have or how many firewall hits, you know, like most people recognize that is not, not useful. And, and I think it's good that that the audience is wider. It's metric— security metrics aren't just for security practitioners anymore. They're for, uh, non-security executives and even the board. You know, there's been tons of, uh, focus lately on what security stuff should be shown to the board, uh, and I don't think that's a solved question. Uh, we've done some research on that, and, um, but the fact that we're having that conversation I think says that we're smarter and the trend is that we've got to become even smarter. And that's a good thing to have that pull.

Yeah, I agree. I mean, I think the maturity level has definitely gone up. Are we where we need to be? No, but we are light years ahead of where we started. And especially for the professionals that have been out there for a while, being thrown headlines that are completely out of context or metrics that just don't seem right, they're going and looking at the source and going, nope, nope, nope, this was pulled out of context, or no, this methodology is Scooby-Doo. Yeah, it's just not going to work, right? It's just not true. It's just marketing fodder.

Agreed. And, you know, and I think to your other question you asked about standardized metrics, I think that's one where, where we really need some work and standardizing that is always odd because there's, it's gotta be, it's just hard to get something that people agree on. You know, you put 10 people in a room and, and ask them what it should be and you're gonna get 10 different answers, kind of like you said earlier. So, but I think we need to be there. To incentivize additional collection and reporting of this kind of information. And the more that's done, things tend to improve. And I take something like, let's go back to the DBIR, right? We collected the information and soon after we produced the VERIS framework, which was, hey, here's information that, data points that we think is useful for if you're doing incident response, try to collect these things. And we can share it. And it was before Stix and, uh, Taxy and some other things came out. And Ferris was never super widely used, but it was the glue that allowed us to take information from 70 different sources and stick it in the same, you know, charts and graphs and stats and, and show that collective view. And, and so if you have that, Uh, it really does raise the bar on what you're able to, to report and do with it, and then, and the meaningfulness of it. And I think things like, you know, MITRE ATT&CK is a useful standard of how we look at, uh, um, uh, threat actor techniques, tactics, techniques, and procedures. Tons of people are using it. It has momentum. More and more reports— I'm doing a study right now where I'm taking every published report that has reported stats around, uh, MITRE ATT&CK techniques and just figuring out, all right, is everybody reporting the same top 10? Uh, why, why are they different? You know, are endpoint vendors seeing the same thing as, uh, you know, cloud, um, uh, monitoring tools that are doing that? And, and it's super interesting, but just the fact that I'm able to do that I think is a positive because that information, they're at least using the same language. So therefore there's potential there for comparable metrics.

Yeah, I mean, that capability wouldn't have been possible years before. And having those frameworks, TAXII, STIX, STIX2 now, I guess, you know, those kinds of things, it helps us be able to grab all that disparate data, unformatted data, and stick it into a format so we can compare apples to apples. So again, another good point that, yeah, our industry is maturing. Not exactly where we're, you know, we're not at the end state. We're not at the, you know, 9th tier of professionalism in our black belt judo of cybersecurity metrics, but we're getting there, you know. And you had mentioned, I mean, your organization does lots of different metrics and you have your pulse on tons of different insights. And I see the reports come out, all sorts of interesting conclusions and the data and the methodology. By the way, I love the fact that you also give the methodology of— right. Yeah. Because again, without that, it's just, hey, I've got a bunch of data that I'm not going to show you. And here's my results. Well, you know, I don't know. I don't know what you were smoking. You know? So, the methodology there is really important and greatly appreciated. So let's talk about maybe some interesting findings, reports around vulnerabilities or exploitations or remediations. What's going on in our industry today that is most interesting to the cybersecurity community here? What do you think?

Lots of stuff. You mentioned vulnerabilities, so I'll take that one. Probably one of the oldest domains in security management. I mean, we've been doing that since pretty much the beginning. I guess maybe crypto is a precursor to vulnerability management, but vulnerability management is probably step 2 in the legacy there, you know. And I, I have found it fascinating. So we've, we've for several years now have, um, been doing, uh, research in the, in the vulnerability management space. It started with, um, Kenna Security, and they, you know, have lots of information on how organizations are, are detecting vulnerabilities and remediating them. Um, and then we have looked at vulnerabilities or flaws, bugs, and code with, with Veracode. On several occasions. We've looked at vulnerabilities, um, and external scans with, uh, companies like RiskRecon and Security Scorecard and so forth. So lots of different types of vulnerabilities. But the cool thing is, uh, in fact, right before this I was on, um, a meeting, and we've been using a technique called survival analysis. I mean, it's a classic statistical technique of, all right, I know, you know, first saw this at some point, saw it at various interval points, and this is the last time I saw it, and how long did whatever that thing is survive? And you see it— they call it survival analysis because a lot of times it's used for, you know, half-life and, and actual survival type stats. But we've been using it in, in the vulnerability management space to figure out, all right, how long does that vulnerability exist in an environment, you know, how long until it's remediated? Uh, and we've been able to apply that, uh, to 6 different datasets in the last few years and compare these curves for different types of organizations, different types of environments, different, uh, uh, severity vulnerabilities and all kinds of stuff. And it's just really, it's really cool to be at the point where we can use those kinds of techniques at different, uh, different datasets and start to, to learn things. But, uh, uh, kind of a long answer, sorry. But vulnerability management space, you know, is there's lots of cool things. So, uh, 10% is a stat that has stuck out in my mind for years that we saw. So the typical organization can only fix 10% of the vulnerabilities it finds in its environment on a typical month. Actually, I think we saw it increase to 15% the last time we did this, but let's just call it 10 to 15%. So there's no way you're getting to them all. And I think that's a fascinating, uh, you know, prioritization. How do we make that decision? Because if we just go and randomly start squashing bugs, uh, we're not gonna fix— we're not gonna fix them all, much less fix the ones we need to. And, um, uh, I think before, uh, we started recording, we were talking about, um, you know, 2% roughly of, uh, known published vulnerabilities have exploits, uh, are actively used and attacks, which tells me that, hey, the purpose is not fixing everything. It's fixing the things that really matter, right? And we've done a lot of research. Can organizations really get those 2%? Yep, they can. And there are clearly strategies that measure better in terms of being more efficient and more comprehensive along those lines. Those lines. And it's just, it's super interesting to take a data-driven approach to a very old security domain like vulnerability management and clearly measure how much better you can be, uh, through certain practices and strategies and, and things of that nature.

And I think that idea right there that you mentioned, right, um, and I remember, I think it was in 2015, Microsoft came out with the original number that I saw that only 5% of vulnerabilities are ever exploited. 2% makes a lot of sense nowadays, if that's what you're telling me the number is, and yet so many security leaders, they make that the very heart of their security program. I am just going to spend my days and my team and my resources closing all vulnerabilities, and then that becomes their goal, and it's just not— it's disconnected from running an efficient and effective program because overspending on security, wasting money is just as bad as underspending. And if we're— your real focus should only be that 2%. Knowing that is key. And I remember reading one of your reports there where you were talking about some of those different methodologies on how do you do that? Do you take the CVSS score and just Gantt chart it, and that's what you're gonna go after. As it turns out, you know, as I recall, you said that probably isn't the best way. Better than throwing a dart.

Well, actually, it's about like throwing it. It was about equal with, uh, with— if you do, if you fix everything CVSS 7 or above, it's about like, you know, uh, uh, it's a little better than, than random chance, I think, but, but not a ton. But yeah, we found that if you just monitor Twitter, I don't know since Twitter's kind of blown up nowadays, but when we did this research, if you monitored Twitter for mentions of a CVE and you just fixed those, it was more effective than fixing the highest CVSS.

Like orders of magnitude more effective? Yes. Or are we talking 15%? Yes.

No, no, no. It was, I don't remember the exact, but it was, yeah, you know, 5 times, 7 times, 10 times, whatever. I mean, it was, it was a lot. Um, so yeah, there's that. And that goes back to how we measure things, how we standardize things. Like, if, if you're going to adopt something like CVSS and that becomes the thing, which it has— I mean, a lot of people have embraced that, it's been out forever— but we're finding out that it's really not very useful. It's really hard to ditch it, you know, and, and move to, to something else. And that's sort of the flip side of, um, this whole metrics thing and, and the, the are we smarter question, right? Um, uh, so that's, that's, that's been a challenge and, and honestly surprising because I would have thought, yeah, you know, when I look at CBSS and what it's doing, it, it makes sense, you know, why— oh, oh, I know this about that vulnerability, so that's that's gonna be higher. But it's not a good decision metric for which vulns should be prioritized for remediation if what you care about is exploitation of those vulnerabilities.

Yeah, and I'm a fan of CVSS, right? And I look at it from a historical perspective. It's like second generation. The first generation prior to CVSS, people would sometimes hear about vulnerabilities, sometimes not. They would kind of do their own assessment, maybe go after the low-hanging fruit. What does that mean? Okay, sure. And it was completely disorganized. And when you would ask people, why are you patching that vulnerability but not that? They'd have a different answer, right? And then CVSS came along and said, okay, well, let's at least attribute some parameters. Is it remotely exploitable? Okay, it's going to go up another 4 points, right? And that made a lot of sense and it was helpful. Head and shoulders above Gen 1 maturity, which just said, eh, do whatever the heck you want, right? And, you know, at that point, I think we kind of stopped and said, that's good enough. It really isn't. Our industry, everything is not good enough over time. It can be the best today, but tomorrow it's a little bit weaker, and next week and next month we have to evolve. And I think that is one of the areas that We failed to evolve. So reports like what you're coming out with is that ugly slap in the face, hit me in the face with a fish and go, hey, you know, it isn't as effective as you think, right? You've got the right motivation, but your execution, and here's the data that proves it.

Yeah. And, and one example of those, which I'm gonna mention just because, uh, I, I like to, to poke a little bit at it is, um, release of exploit code. You know, there are passionate security researchers that think they are saving the world and, and doing great things by finding vulnerabilities and publishing all the information about them and writing proof of concepts and to demonstrate how it can be exploited and all of that kind of thing. And I think genuinely thinking they're doing a good job, but I mean, we've done a lot of thorough analysis, and, um, when exploit code is published before the patch is available, it is demonstrably a negative impact on defenders being able to remediate before exploitation occurs. I think it like sets them back 6 months or something like that, if I, if I recall correctly. And that's a hard conversation because as soon as you say something like that, some people, I mean, they go like right to religion mode, you know what I'm saying? Like, it, it's, it's, it gets uncomfortable. But I think that's an— it's an example of an area where we got to ask the questions. Okay, we're not saying exploit code is bad, never has any uses, you're a bad person for doing it, but we gotta look at when it's published relative to when a patch exists and how that enables people to deploy patches before exploit exploitation hits critical mass. And, you know, let's study these things and make some better decisions. I think, I think we can improve the way that we, that we roll those things out. Um, so yeah, I'll, I'll, uh, I'll get off the, the soapbox on that one.

Well, no, I'm gonna, I'm gonna join you on that soapbox because that was, again, it was a religious argument for, for many, many years. And I can see both sides and I'm actually kind of happy with where the industry moved The companies that were originally suing white hats, you know, or vulnerability researchers for discovering something and putting gag orders on them, have backed off and instead embraced vulnerability programs, bug bounties, all those kinds of things to recognize reward.

Yeah, all good things.

Love that. And you had this other extremist group going, we have to slap the industry. If we release the exploit code, forces them to acknowledge and create a patch because bad things will happen. And I think they've pulled in a little bit, not all of them, but most of them have.

I agree.

And now we've kind of settled in this, there is a responsible way of reporting these.

Yep.

And there is financial incentive to do the right thing versus selling them to the bad people on the black markets. So, you know, or the underground markets. So I like where we've come from the extremes and most organizations, I think, have kind of settled on, okay, let's support responsible disclosure.

Yep. Yep. And I expect that definition, what is most responsible, to shift. I would say, hey, let's make sure that comes out after the patch, for instance. You know, is the most responsible based on information I've seen. So yeah, I, but I love those kind of conversations and I love the fact that we have data to actually analyze them instead of just have an opinion war.

And I'll go off on a tangent here because I'm seeing that exact same discussion unfold with like ChatGPT, right? AI GPT systems where people are coming out and saying, hey, I got around the security controls, I got it to write malware, I got it to do this, I got it to do that. And there is no sense at all yet of what responsible disclosure is because everybody's rushing towards it and they want the fame, the notoriety, they want to show how smart they are, they want to do all these great things. And yet the bad guys are just sitting there going, really, I can do that? Let me go automate my phishing campaign to be 500% more effective. Oh, well, that wasn't really meant to to do. Be careful, your words matter and your innovations matter. So I'm seeing that happen again and I think we will see it again with the next major technology and the one after that. It's—

Yeah, it's crazy.

It is. Okay, so what else other than vulnerability disclosures and managing, what are some interesting data or reports that have just kind of knocked your socks off in the past year or so?

So, you know, we've been doing a lot of emphasis on cyber risk quantification. You know, I think that is one of the conversations. A lot of it came from metrics. Okay, now we can start measuring things and measuring, you know, security in dollar terms has been sort of a desire for a long time. Some people think it's impossible. Some people think it's—

Turn on security investment, baby!

Maybe, maybe sort of, kind of, but You know, this, it's actually picking up steam and, and, and it's cool to see that. And so we've, we've put a lot of effort there and, and I think done some, some really good work in terms of putting the frequency side and the losses side that you need for risk quantification together. Uh, we have a study series, it's free if anybody wants to check it out, the Information Risk Insights Study. If you go to scientia.com/iris, IRIS, you'll find several studies, all of which are analyzing historically reported security incidents and trying to understand, well, okay, well, what's the probability of an organization like yours in your industry of your approximate size is going to have some kind of cyber event in a year? You know, what if that happens? What are the losses going to look like? Like, we can measure that. Now, the curve of losses is like from $100 to multiple billions, but the fact that we can draw a curve and create some parameters is, I think, really, really interesting. And we have sufficient data to be able to do that. And I think that's going to improve decisions in the way we assess risk and hopefully defend our organizations.

So I've got a request, but before I go to that, I wanted to know, how you think AI, ML, DL, and even GPT type of AI technologies, how are they going to impact cybersecurity metrics either in data gathering, analysis, generating information in understandable bytes, you know, accuracy? What do you think is gonna be the impact?

Uh, I'm, I'm very curious to know, and I've heard a lot of people say, oh no, is this gonna like replace my job or, or security jobs? And, and sometimes I actually hope so. Like, I took, um, some findings, just a finding from a study that we had, and I said, all right, uh, we just did some research and we discovered that X. Write 3 paragraphs that describe that finding and why it's important and offer some recommendations. And, it did a pretty damn good job. And, I mean, my impression was like, oh, thank God. You know, I can— I know exactly.

Now I don't have to write—

if it can write content for me, that would be amazing. I realize some people probably have different, different views on that, but—

sure, sure.

You know, but I think there are some, maybe some help there. I think the information collection is super interesting. So, I plan to do some experimentation on researching public events. So, if we're— I mentioned cyber risk quantification. You know, we've— some of those studies, we took like the 100 largest cybersecurity events in the last 5 years and collected all the information that we could on them. You know, every data point possible. I think it could speed that process up so that we could do things like that. Not— it took us months to get the 100, you know, maybe, maybe we could do 1,000 realistically in, in a month or two and, and really expand that study and learn even more. So, uh, and that would be, that would be amazing if it— and you can't trust everything, that's, that's the problem, like, but I think it's getting better and we'll figure out what things it's pretty good at and what things it's not. But I, I have a genuine hope that it's going to be faster at finding disparate public information spread out there than I am on things that are known. And that's going to be useful.

Yeah, I agree. And, you know, I get lots of people asking, you know, hey, is ChatGPT going to, you know, replace people? Is my job going to go away? And my answer is always, your job is going to get replaced, but it's going to get replaced by a person that knows how to use ChatGPT. So if you want to keep your job, you better start learning all the new powerful tools, right? If you were on a typewriter and Microsoft Word came out, you better learn how to use Word. This is the same thing. Don't be afraid of it. It's not black and white. It's simply another tool. And I think people in your industry will be able to leverage it as well.

And I think there's a lot of parallels to like the early days of the internet. You know, where, you know, now you just, you wouldn't be very effective at your job if you couldn't search up information that's out there.

Okay, so let's go back to my request because you were talking about data, you were talking about, you know, returns. So my request in the future, because I'm not sure if it's even possible now, how about you tell me if it's possible. Can we create metrics or pulled together data and derived the proper measures to create metrics and information that tell us what value propositions security can be across preventing loss, across being, providing or opening, right, enabling the business. Let's just choose that as the next thing. Third, being a competitive advantage for some companies, some industries, and lastly, for actually generating new revenue, um, as a benefit, right? As consumers and everybody become more savvy, becomes more of a decision criteria for products and services and, and whatnot. Do you think we're there yet? Do you think metrics where we're at now can deliver that and be able to provide that type of information to the industry. Because I— that's the one thing that I think people want to be able to bring to the board. And that one— that aspect is so relevant to most of the businesses, all the businesses, and most of the organizations and even governments out there, that it is that next powerful piece of data. That's— we're bumping up in black belts, right? We're moving to the 8th degree. Is it possible? And if not, when? If?

I think bits and pieces are possible. You know, there's some things in there about security adding to revenue that are probably closer in certain types of businesses. Than others. And that's, you know, that's expected. That's okay. But I've seen those things studied. I mean, you mentioned enabling the business. You know, we did this. This was survey-based. We don't do a ton of survey-based research, but we did a study with Cisco, and they did the survey right. It was like they farmed it out to a professional survey firm., YouGov or something that did stratified random sampling and did it right and got 5,000 responses. And, um, so it was, it, that was good. We analyzed the data and, uh, enabling the bi— there, there we were looking at all kinds of security outcomes. It was a security outcome study or security outcomes report. Um, and enabling the business was one of the, the outcomes that, that we were interested in, in addition to avoiding security incidents. And the sort of more traditional security stuff. And there was correlation. Again, it's a survey, but we asked, you know, several things and we started to see some correlation to, yeah, you know, there are things that correlate with enabling the business, at least people, you know, people reporting that their organization is doing a good job at that. And I think, you know, things that we can measure internally are improving. Uh, you know, I, I very much want the first part of what you were talking about, about, you know, uh, can we show reduced losses and events and other things? You know, I, I, that, that's one of my career goals is to be able to, uh, study, you know, a sufficient quantity of organizations over a period of time and be able to tie some type of organizations that do these things are X percent less likely to have incidents and, you know, X losses or, you know, whatever, because that's super hard to measure. I mean, it's ridiculous that it is that hard to measure, but, and I want to do that in a real way, not a, not in an isolated lab or something like that and show, hey, look, this blocked all these attacks. But really in the process, the running of an actual organization, an actual security program, can we measure efficacy, you know, and start reporting on that is, I think we're getting closer. That's awesome.

That's positive news. Okay. So last question. You know, we've talked about how the maturity has grown. With cybersecurity metrics. We talked about how it can change people's perspectives and help them with prioritization and efficiency and define good goals and, and all of those things. So cybersecurity metrics is important, um, and it's meaningful. And with any kind of tool like that, we go back to what we kind of touched on a little bit before around the ethics of cybersecurity metrics. And we have seen data be used to flame fears and create uncertainty to drive doubt right into the minds of consumers and businesses. Has the industry moved away from fear-based models? And where do you kind of see our trajectory here?

No, unfortunately, I don't think it's moved away from. I, we might be past peak fear. Like I sort of consider the height of the APT era to be the peak where, you know, it's like just hammering on everything is hopeless.

Cyber Pearl Harbor.

Right, right. So maybe we're past that and, and little better ground. But the bottom line is that sells and it probably sells security product better since we can't actually show effectiveness of various, you know, security controls and practices and products very well. We gotta resort to the fear card. Card. And, uh, I don't, I don't love that, but I understand why it's done. And also when you play that card, you tend to get picked up by media and other things because they view that as news. Um, and so it's, it's a cycle. Um, and, and your base question of, um, ethics in metrics. I, again, overall, I think there's improvement there, you know. I think, um, there's, there's a, there's a lot of people in the industry that really care about what they're doing, and they're trying to report, uh, on whatever it is that they're doing and put that information out there. And I think a lot of it, especially from the researcher community, comes from, comes from a, the right place that wants to produce something useful. I think sometimes that is hijacked by marketing and PR teams, you know, and packaged in a form where the reliability and ethics of it is tarnished. But, and hopefully we'd get better, we get better at that because I'm, what I'm hoping is those teams will learn that. Your target audience responds better if you don't dilute it, if you don't, you know, misuse that information. And so there's incentive not to. And so I think I'm, you know, I'm, I'm optimistic in, in that sense.

And you burn somebody, you lose your trust. Are they gonna believe you the next time? You know, selling fear, I think, does work. But I think there's a cap on it because as soon as you do lose, as soon as they realize, yeah, I've been played, right? They're not gonna come back to you. And so for an individual practitioner, it's nothing. I, I, I never recommend doing that. Come in with data, be realistic, have the story, bring the metrics you can and, and say what you know and what you think, right? But if you're just using fear, if you're using the FUD, fear, uncertainty, and doubt, It'll work a little while, guaranteed it will. Sky's falling. But then you're going to be replaced, right? Because as soon as they don't trust you, your value goes down.

And unfortunately, if they don't trust you, they don't trust me either because I'm just grouped into the, oh yeah, security industry produces a bunch of junk, self-serving junk. Just ignore it all. And that's really unfortunate. Encouraging it because there's some good stuff out there like we talked about earlier. Yeah.

Yeah. Well, thank you for your contributions to continuing to produce that good stuff.

Likewise.

And I have to say, you know, the Cyenthe Institute, right? You know, to the people watching, go out there. You're going to find some of the best analysis, metrics, research report in all of cybersecurity. We'll put the link in the description down below. And thank you, Wade, for coming and talking.

Thank you so much. I appreciate it.

I'm going to reach out to you again to talk about new reports and all sorts of stuff and reminisce about the old days as well. And thank you all for watching. Be sure to subscribe and catch all the Cybersecurity Vault episodes where we chat with industry leaders like Wade to dive into the most relevant and interesting cybersecurity challenges, perspectives, and best practices. We'll see you next time.