Podcast S01 E05: Assessing Institutional Risk Management Strategy for Threat Intel Matt Devost OODA

All right, thank you guys for coming on to the Shadow Dragon podcast today. I've got Matt Devo and to. And we are going to go over a bunch of different things, but Matt has been in the industry for a very long time. He's been on helping out with Black Hat, Devcon, iDefense, Fusion X. I mean, you've got a slew of things that you've had success in. Let's just get to it. You know, what, what are some things that you're working on? You know, what are, what are the things in the industry that you're seeing that are awesome and not awesome? And then let's, let's jump into the topic of disinformation. What is it? What can we, you know, grasp about it? How, what's the history on it, and what does it mean for, you know, information security professionals or just, you know, your mom and dad reading the news, you know, and we'll follow up with the inbox, the Q and A for the day and just start kick stuff off here.

Excellent. Thanks for having me on the podcast.

Awesome. Yeah. And so for those that don't know, I've known Matt a little bit for a few years now, and we've kind of intersected a few different times. And Matt's always just had a really good team around him. Always been pushing the envelope and kind of the guy behind the scenes. For those that don't know.

Excellent. Yeah, a large variety of things. You know, I got my start as a white hat hacker in the DoD and Intel community back when you got to have a lot of different firsts. Right. So it was kind of the first time that type of team had been stood up. First time we were authorized to go after classified and unclassified systems, and then also had the first time operating in a coalition environment. So we would stand up a red team during classified military exercises and it would expand the portfolio of companies that we could target from just US Systems to include the five eyes. So Australia, New Zealand, Canada, UK and the United States, given the time frame, you know, get to rack up a lot of different firsts with regards to being first person to get onto systems in an aircraft carrier while deployed at sea, nuclear submarine, you know, kind of all that sort of excitement, and then transitioned into a bit of an entrepreneurial role. One thing that was interesting, you know, by way of a cyber threat intelligence perspective, is because I was a guy who could be hands on keyboard, but also had a master's degree in political science with a national security studies focus. I understood the Kind of policy and national security implications of all this stuff. And as a result I got pulled in to support the President's Commission on Critical Infrastructure Protection. If you're not familiar with that, it was a mid-1990s initiative, a presidential commission focused on looking at what are the risks to our critical infrastructure with an incredibly heavy focus on the cyber risk. So I got to write a good portion of that report. Wrote an infamous report actually to try and educate the White House and the commissioners called the Hacker Primer, which talked about hackers are not evil people. Because I had this unique distinction of being a guy with a TSSCI security clearance, but still going to DEF CON and summercon and all of these events and trying to bridge those two worlds out of PSIP. There was a Presidential Decision Directive 63 that was issued and for those unfamiliar, that was actually the directive that established the ISAC system. So information sharing and analysis centers and seeing the need for cyber threat intelligence and not wanting to wait for the government to do it, a group of us got together and stood up high. Defense is really the first cyber threat intelligence company that launched for me a career of entrepreneurship over the years that included not only working on that company, but I helped, you know, the startup called Security Design International. Ran my own company, the Terrorism Research center that despite the name, did a lot more than just terrorism related research with a global fusion center that provided 24. 7 intelligence for private and public sector clients. Had an active assessment program, cyber and physical, and then did a lot of interesting stuff for the US government. Did that for 13 years through acquisition at the 10 year mark and three years post acquisition and then helped start Eyesight Partner. Not start, but helped build up ISIGHT Partners. John Waters had reached out to me and wanted some help and went in there as the director of Operations during kind of a key time in their growth. That transitioned to FusionX, which I think was the first time we met in 2010. And FusionX provided just very specialized red teaming and we'll call it kind of threat investigations, threat attribution type support for commercial only customers. And we sold that company in 2015 to Accenture and I stayed on an Accenture for a little over two years running their global cyber defense practice.

Yeah, see, I'm telling you the guy behind the guy. Right. I remember too, I mean, going back into the critical infrastructure stuff, that was pretty pre. That was when Nipsey was still starting. Right?

It was actually, yeah. It was the kind of the catalyst for that to be formed as well. Right. So the government kind of, you know, had the hallelujah moment around critical infrastructure protection and the nexus to cyber. So you had the National Infrastructure Protection center getting stood up. You had, you know, law enforcement, military intelligence community getting much more interested in this topic. Defense Science Board a few years later did a, you know, defense and base oriented focus on recognizing what infrastructures they were dependent on. So it turns out, you know, your bases are not self sustaining entities as it relates to power and telecommunications. And as we know, a majority of the communications, especially for things like logistics and workflow, flow over unclassified Internet pipes. So it was definitely an interesting moment in time where the world finally caught up with a problem I had been writing about since 1992 in that this nexus of, you know, dependency on technology, that technology being inherently vulnerable, was creating a unique national security issue.

Yeah, yeah. I mean, yeah, then that's a long, long. There's a lot to pull out of that. I mean, so just from the nipsey stuff, then you had, you know, some of the infragard stuff spawn out of that.

Yeah.

Then now you've got, you know, the FBI running a lot of their, you know, you know, I guess their, their citizens academies coming out of that. I mean, so there's a lot of, you know, eggs getting hatched out of that over the last 20 years. And, and for, for some of you guys, that absolutely. Yeah, go for it.

Yeah. I was just gonna say, you know, speaking of the FBI, I mean, another thing I did in the kind of mid-1990s is I presented the first ever cyber investigations course for the FBI. And they pulled people in from all of the different field offices. And that was a point where they had their cyber ninjas. And for them, the definition of cyber ninja was somebody who had a new laptop. It wasn't to do with anything with skills. And remember stepping them through the value of open source. And we did some live demos on doing pager monitoring, because back in the day, law enforcement very dependent on pagers. And if you recall, it was very easy to snag those signals over the airwaves and decode them. So I showed how I could de. Anonymize somebody just based on having their pager number. And I still remember to this day, I spent the whole day giving this instruction. And then a lawyer stood up and said, hey, by the way, you can't do any of the stuff that Matt just told you about because the laws for how they deal with open sources and that type of stuff had not been kind of fully fleshed out yet. So I Definitely had this moment, like, well, why am I here? I guess it's for informational purposes only.

Yeah, yeah, yeah. And then they're like, now this is all classified. And you're like, what?

Yeah, you just can't do this. It's great to know that this capability exists, but you're not allowed to do it.

Right, Right. Matt, we've got Brian that just joined on here.

Hey, Matt, how's it going?

And just I. I'm sure you guys have intersected in that same realm because Brian was doing a lot of the. The teaching over at the bureau for their cyber classes in the 90s.

Probably, you guys, 98 through about what, 2006. 7.

Excellent. Yeah, that's probably the possible. We met at some point in that capacity.

Yeah, you're absolutely right about those early days, though. But it was just, it was. It was rough. We were actually teaching basic Internet usage at the time because, yeah, you had to start them at that ground level. You could not assume that everybody actually knew anything about a computer or the Internet. Stuff like protocol, stuff like that. It's crazy. And they were expected to do investigations.

Yeah, yeah.

So. So, Matt, you know, kind of just, you know, looking over all the stuff that you've had your hands in, one of the first things that I kind of have. My question for you, just growing through that, how have you remained fresh and maybe not gotten too jaded or what's the tension with becoming jaded with how the industry's grown and not grown, because, you know, just full disclosure for me, I'm at the jaded point now. You know, you know, after 20 years, you know, anybody knows me knows that, like, I just. Anything that becomes a cultural norm, I just start becoming countercultural on it. And, you know, I. I'm kind of tired of the way DEFCON is so huge now. I liked it small. And, you know, how do you deal with some of those tensions, with moving through some of your history and where you're at now?

What.

What's some recommendations that you can provide?

Yeah, I've always tried to maintain a positive outlook for me, especially as it relates to kind of the security community. It really is. Yes, There might be 25,000 people at Defcon, but there's still the several dozen that I know. I'm going to sit around on the floor drinking Red Bull, talking until three in the morning about really interesting stuff.

Right, right.

So that's kind of fun. And then, you know, getting involved in other initiatives. I'm on the review committee for our local B sides. Right. And participating in some of the more grassroots initiatives with Shmoocon here in the D.C. area, although that's grown so large that I can't get a ticket to attend anymore. Gotta go run the lottery system. But you know, I focus a lot of my effort on a, you know, where I reach points of frustration is people that kind of forget the past. And I allude to this on social media every once in a while. I get tired when somebody thinks that they've discovered a new threat initiative or discovered a new technology and it's something that is 20 years old. Right. So I do a lot of preaching around understanding the history of the community. Right. Come in and fierce domain or you know, some of the stuff that talks about where we've come from over the past 25 years and some of the ideas that we've had. Cyber Manhattan Project. Yep. Been to six of them. Right. I mean, it's not a new idea. Information sharing and analysis, public private sector cooperation, not new ideas. Don't prevent, don't pretend like you're reinventing it. So try and balance out that, you know, kind of the folks that are out there articulating that, you know, they've invented the latest and greatest and bring that historical context to it. And I get pulled in from a government perspective on those issues as well. I've been dealing with state sponsored threats for over 20 years. History informs the future and we did a really good job over the years of predicting where this environment was going. So try and spend a lot of time working with them to try and figure out, well, where are we going over the next five, 10, 20 years? And for me, that's where I get most excited. The other kind of big initiative for me is, and I alluded to this at the B sides keynote I gave like a year and a half ago, is that it would be a shame if the hacker mindset only ever gets applied to computer security problems.

Yeah, right, yeah.

That ability to dissect issues, take them apart, make things better, speak truth to power. We should be bringing that mindset to lots of different aspects of society. So I've tried to be a little bit more active in advising political leaders and thinking through issues as I see new technologies. One thing I write and talk a lot about is artificial intelligence.

Right.

We don't have real AI right now. You know, in the cybersecurity industry you get a little frustrated because you get this head fake of AI and everybody says AI, but then they pass the ball to machine learning and you're like, well, that was Kind of boring.

Yeah, yeah. I always ask people and I'm like, hey, so how long have you liked linear algebra? And they're like, it's like. And I'm like, I see what we're talking about here.

You know, that said, these data science initiatives and machine learning and this march towards, you know, almost a real type of AI, or at least, you know, some narrow AI implementations are critical. Right. We're going to be spending trillions of dollars, they're going to enable finance and telecommunication, all these different businesses. I look at it and say, why are we making the same mistake with the development of this technology that we made with, for example, just the different development of the commercial Internet in general? Right. We didn't think about security from the get go. So I do a lot of evangelizing about saying, hey, let's build security frameworks in from the beginning, let's articulate that security is important from the beginning so that we don't deploy these technologies and then all of a sudden have these issues associated with them. And I'll tell you what, where you have something where it's incremental machine learning, it is like having compounding interest in a bank. If you have a mistake early on in the algorithm, it just learns to be worse over time. And the only way to revert is to kind of go back to a state zero. So you have this potential for investing hundreds of millions of dollars in your AI initiatives in your corporation, having the algorithms develop some bad aspects to them and then having to regress back. And these are all problems that can be solved. You can take a red teaming approach and look at these issues and provide some frameworks and. But I get frustrated when, when I see the technologies being deployed and those frameworks not being incorporated.

Yeah, yeah, I just heard about an example of that. They were using machine learning for looking at chest X rays, which I thought was a wonderful thing. Right? Chest X rays, they have zillions of those. And the way they'd applied it, what they didn't realize that the, the system was actually looking at a little note on the side of the chest X ray. And it could tell the difference between chest X rays generated at say, a small doctor's office location from those machines versus one done at a hospital. And the algorithm learned to recognize that, well, there's probably a higher probability of there being an actual disease on a frame generated from a hospital than there is at, at a small office. So it was weighting those more heavily and of course creating tons of false positives because that was what it was looking at. They deployed this thing without understanding that. So it wasn't until they went backwards, ripped the thing apart and went, wait a minute, it's just reading the code on the side of the X ray.

Yeah, yeah. No, it would be good to kind of frame those issues in advance. Right. You know, at least get somebody to red team it and say, well, you know, hey, this is also data you're providing on the X ray.

Right. That assumption that, you know, the computer doesn't look at that.

Yeah, I read the similar type article with an analyzing EKG results where they felt like the algorithms were able to predict when somebody was going to die, but they didn't understand why. Well, that's kind of maybe an important.

Thing that we don't let despair get in there too much.

Right? Yeah.

Now you guys do this Turing integrity assessment, which I've never seen anybody advertise anything like that at all. What's that about?

Yeah, it really is just, as I mentioned, trying to bring a framework for thinking through these AI deployments with the lessons learned from cybersecurity over the past 20 years. It's a combination of looking at it from the perspective of the robustness of your algorithms, taking a red team or mindset of looking at how might these algorithms fail? What data dependencies do they have that they might misinterpret, like with your example. So looking at it from that kind of red teaming data science perspective, looking at the integrity of the training data, again from your example, but we see over and over that bad data in or that these systems will inherit the biases of the data that they have. So looking at the robustness of the training data and then also looking at the security of the infrastructure that it's built on, which is just classic red teaming. Let's not deploy these technologies on insecure systems. And then the last piece, which is a little bit more difficult and nuanced, is what does an adversarial approach to your machine learning look like? If you have built the tool that does sentiment analysis of Twitter to try and determine when your hedge fund is going to buy stocks, it's great at filtering out. The guy who went and created 5,000 Twitter handles and all in the same day said, apple sucks. Okay, it's pretty easy algorithm to write. How would it do if a nation state did something around an influence campaign type of strategy where they spent years, they integrated those accounts into the ecosystem of social media and then started propagating certain messaging? How would you deal with that? How would you Deal with the visual analysis algorithm that gets fed bad data. And we see examples of this. It seems like on almost a weekly basis where somebody trains an AI that's supposed to be identifying dogs to identify crocodiles as dogs or whatever it may be. So looking at that again, my specialty over 25 years has been red teaming. So I tend to apply a red teaming lens to a lot of these issues. So it's saying, let's apply a red teaming and cybersecurity framework to how we deploy these technologies.

That makes a lot of sense. Along that same line, was it we just said, well, so excited pivot off that was. We just had a friend shopping, you know, new company around to VCs and stuff like this and he, you know, do his pitch deck so forth and, and he did six in like two days. It was awful, I'm sure. But at the end of it he's like, they would always ask me the same question, like, well, where's your ML in this product? And he's like, well, it doesn't rely on that. It's the wrong sort of tool for that sort of solution. As like they were just like, nope, no ML, no money, we're out. And so I think it's the rhetoric is hurting the industry. But let me get your opinion on that.

Yeah, yeah, I mean it absolutely is. I mean, I'm sure, you know, still go out to rf, rsa and two years ago you couldn't escape machine learning and AI at rsa. And then the market starts to push back.

Right.

What I see is there's still opportunities out there for smart money around some of these approaches. I do a lot of angel investing. We're building a small seed fund. The whole idea being, okay, we can identify real opportunities and do the real technical due diligence and not come up with a stupid qualifier like that, hey, not doing machine learning, you don't get our investment dollars or you can build an AI component into this. Right. The reality is that the applicability is going to be determined based on the solution and you got to be able to distill that out from the opportunity itself.

Yeah, yeah, I remember we went and helped out some investors at RSA a few years ago and it was just that everybody was on the AI bandwagon and you start asking any kind of small question and pretty soon you're differentiating. This isn't AI, this is a fast four year transform or don't try to pull the wool over my eyes. And some of the people would just be like, oh my Gosh, how do you even know that? I'm like, well, a lot of this is just snake oil. And that's hard. Talking to some of the clients that we've had over the last few, few years. Like, one of the quotes that stood out to me the most was a guy said, you know, it's kind of strange, like, when I'm around you information security people, it just seems like it's one magician with tricks versus another magician with tricks. And, you know, he wasn't technical at all, but, you know, the way that he articulated that was really interesting because, you know, you. It's hard to see, like, what's true, what's real from the investment side or from the client side. Sometimes when they may be overwhelmed with obscure knowledge or overwhelming facts that they just don't have the calories to. It's too many calories to consume. And it's just.

Luckily, in the cybersecurity industry, the buyer community is getting a little bit more sophisticated. If I look at some of the CISOs now, friends of ours that we consider to be hackers and cybersecurity folks now starting to be into those roles, asking those tough questions and kind of pushing back in the market a little bit. But the higher up the chain you go again, the more education is needed. Right at the board level. A lot of confusion. I've had boards of directors say, hey, I feel like I'm being held technologically hostage by RCISO because I don't understand a thing that they say. And my response is always, well, why don't you invite somebody on the board that understands cybersecurity? Is cybersecurity important to your business? Is it one of your top priorities? Why is there no one on the board that has that representation? Starbucks has a board of directors member who's a millennial, because the millennial market is important to Starbucks, and they want that perspective. We haven't seen cyber kind of elevate to that level. So you, you get this educated CISO layer, but then you start to go above them and they start to encounter resistance and you start to have issues around. You know, I, I read that AI is in cybersecurity now. How come I don't have any of that?

Right, right.

And you're just like, whoa, when the reality is, yeah, you could not point to an AI solution in cyber security right now. Is there some, some, some interesting machine learning, data science types approaches? Yes, absolutely. Is there some stuff you can do on pre processing and data normalization? Yes, absolutely. But is anybody doing A.I.

No, yeah, yeah, exactly. I mean, I've, I've even been framing it as, you know, there's some of the things most people would call image recognition. It's just, I, I'm just saying object recognition, it's, it's. There's some object extraction out of some images that you may be able to get some valuable information on and work on. Sure. That, you know, is that AI. I mean, you know what I mean? Like, it's so polluted now, you know.

Yeah, yeah, absolutely.

So I wanted to ask you too, Matt, on kind of your, your thoughts on when you were starting iDefense. This is. This was pre. The big wave of threat intelligence that we kind of all got caught up in a little bit in 2010, 2011, even till now. I'm starting to see kind of the tide shifting and what people are really wanting to spend money on and what they want to define as value in that space. But what are some good contrast between the idefense time when you first started it, the middle of the road time when this wave hit, and kind of where we are right now to kind of give, you know, some of the board, board of directors, some, you know, some insights, some CISOs, some insights, and maybe some of the newer guys on the block, you know, what's in the future and contrast across all those different perspectives because there's not a lot of people that have been in all those places over the years.

Yeah, no, absolutely. So, I mean, you know, I still consider, unless somebody proves me wrong, that idefense was really the first cyber threat intelligence company and nobody was doing it 20 years ago when the company was started, the market shifted pretty quickly once the ISAC started getting formed. But back then it really was about a couple of different variables. One was just access to intelligence. Nobody was collecting this intelligence and providing it to these companies. And I remember sitting down with the chief Security officer, Chief Information Security Officer, which title he held for Citibank, and saying, hey, here's the report that we're putting together. It was not anything fancy. It was the 15 to 20 things you need to know if you subscribe, you're going to get access to this information on a daily basis. And just getting access to the information was catalyst enough for people to start subscribing. Then we shifted into the threats got more complex. The Internet expanded into different geographies and different languages, and the threat intelligence started to focus more on being a collection type activity. Right now I needed translators, I needed unique collection capability, I needed unique technical capability, and now it was about aggregating from lots of different sources to kind of get that picture. And I think the third wave that we're kind of in right now is the contextual threat intelligence age. Right. The companies want to know the. So what, what does this mean? And how can I take what you're providing me and actually engage in actions and in my organization that benefit my cybersecurity posture? Right, right. So it's not enough to just throw a ton of data at them. Some are sophisticated enough that they can drive that decision making process on their own, but for a bulk of the market, they want that context. What does this mean? How does it improve my risk? How do I manage, how do I mitigate, you know, how can we drive actual actions out of the threat intelligence that I'm receiving?

Yeah, yeah. I mean, and we've kind of fit in a very specific niche space in that regards with Shadow Dragon, with, we're just providing collection, you know, we're not going to try to provide any observations than anything else driven by the customer, which is really kind of, it's been a strange place over the years because there's that, you know, was it threat intelligence, is it a feed, is it malware this and malware that? And I was like, well, we have some malware stuff, but all of it is in the philosophical vein of you need to make the observation. So, you know, like hitting that mark in the marketplaces has been interesting because we get to see like, who are some of those advanced teams, you know, in government and commercial and, and how does that really look like and how are they growing and how are we growing with them? But yeah, I agree completely with you, with you on that. Just because even some, like some of our customers that are resellers are starting to do professional services on top of our tools. Well, we just want to do collection. We just, we need them to make those observations because we have to be agnostic.

You know, the collection piece never goes away. Right? I think, yes. And you, I would put you guys in kind of in the unique category, right on the collection side where you're surfacing stuff that is unique to the process. And I'm sure you get feedback, right? The first feedback loop that you get is does the customer retain their subscription or not? Are you getting new customers? There's that kind of feedback loop just based on the economics in the marketplace. But then I'm sure there are other times when they say, hey, this was really useful, or this source is useful or can you do this? You're incorporating some of that feedback into your processes as well. But yet, at the end of the day, somebody needs to be making a decision based on the, the intelligence, whether that's a consultant or whether they have a sophisticated enough intelligence cycle within their organization to do that on their own. But I really like it when you see organizations that have a security entity that is intelligence driven, where they are making decisions based on real intelligence. If we think about just risk management in general, it's been a very simple formula for 50 years. It's the, what are the threats to my organization? So I need to understand the threats. What vulnerabilities do I have within my organizations? They need that awareness, what their security posture looks like. What's the impact? We see a lot of organizations now trying to balance out. Well, do I need to manage all of the risks to the entity or do I only need to manage the risks that would be catastrophic or consequential to me. So doing a little bit of triage and not trying to protect everything, and then you marry that together to try and have a risk management strategy that is kind of fit to who you are and what you do and how you're being targeted. Those, I think, are the most successful security organizations that we see out there.

Yeah, yeah, I agree. I would say one of our clients that we've been working with for a long time, they had kind of given us a picture of, instead of doing whack a molecule, you know, you know, taking, doing takedowns on all these sites, they, they had done the analysis that the, the takedowns don't actually do anything. In the end, I think this is probably going to come out a little bit more in, in, in news and in publication here in the next year. But they were saying, like, look, you know, this takedown stuff just, it's bs. I mean, they have enough money to like analyze everything and buy everything and then come to a conclusion. And the flip side that they did was they took a strategic, a strategic approach against some of the top actors that are developing tools against their brand. And then they take a year and a half to take those five actors out, get them arrested, hand that over to law enforcement, do all the process on that, disrupt that ecosystem, and that actually contributed to 20% of the fraud. Yeah, you know, so it's, it's a thinking problem more than a, you know, we got this threat stream or we've got this thing that got taken down, or these stats that we can just give to the board that may contextually just be B.S.

Yeah, that's important, you know, that's an important distinction is it's, you know, famous Jack Welch quote that you get the behavior that you measure and reward, right? If you're, if you're measuring your success as how many takedowns did I issue last week as opposed to am I actually reducing the risk in any sort of meaningful way, then that's the type of activity you're going to pursue. If you say, hey, I want to disrupt this ecosystem and I want to disrupt the availability of tools and thus impact the trickle down aspect of it, because we know how this works in the ecosystem. You have the really smart folks that build the tooling and then they sell that and then you have the actors that operate on the exploitation side. If you can go upstream, you can, as the experience of your client, have a more measurable impact on the security of your organization than if you engage in that whack. A mole. How many sites can I hit? Red teaming went through the same type of process in the 90s. When I would brief a red team it would be hey, you had 50 high level vulnerabilities and 75 medium and 100 low. And if I went and talked to you the next month and it was hey, you only have 45 high level vulnerabilities, you would view that as progress. And in one sense it is. But I can't show that in removing those five high level vulnerabilities that I've removed the right five, the five that are likely to be exploited, that are going after systems that would be meaningful or impactful to me. Right? So now it's a matter of, you know, when we did red teaming and you know this from the work we did together when I was at Fusion X, it's we go brief the board and say here are the threat actors that are likely to target you. Here are the goals that they're trying to achieve. Here is the attack surface that presented itself. Here is the outcome that we were able to achieve. We stole money from the bank or got access to your ip. And then here's the mitigation strategy that is a lot more meaningful of an approach from a security perspective than the old way of briefing number of vulnerabilities, metrics and measurement matter in this field for certain.

It's very human though to want to go that way. You want some kind of reward. And I think non technical people that are dealing with this, you know, from a board level or wherever it may be, they're like, please just, just let there be something I can grab a hold of. And go, that was a success, you know, and, you know, and they. So they tend to push their tool set and everything else around they do about anything that I can measure that looks like we're making improvement. Let's make that happen. Even if it's not the right thing.

Sure.

Well, yeah.

And I think. And you still want to manage the total vulnerabilities in the environment and things of that sort. It's just making sure that from a risk management perspective, you're measuring things that are important. When I would get asked by executives or boards of directors if there was a single thing that we could measure that is most meaningful, I would always tell them time to detection. How quickly, you know, we're not going to keep somebody out of your network. How quickly do you detect that you have an internal problem and then how efficient is your response to that? If you have a six month or six week time to detection, it's game over. Right. If you can compress that time frame, you're imposing a cost on the attacker. They're either going to get caught, they have to move quicker or more dangerous. Maybe they're not able to target the systems they want to go after. So I'd always try and drive them towards, well, knowing that every attacker is an insider at some point. How quickly do you detect that you have an issue on your systems or within your network?

Mm, yeah, that one thing question, that's, that's classic. I get that all the time. So what's the one thing? And I think it's oftentimes folks trying to buy their way out of security. Right? Like just, just tell me the name of one product, I'll buy that and it'll work. And then I'll never have to talk to these cyber security guys again.

Yeah, there are no silver bullets, but there definitely are, you know, things that matter. And there's an example where threat intel comes into play in. And I'll use an example of one that Daniel and I worked together on years ago where a board of directors brought myself in and a couple other folks because they had this issue where almost every year on the year they were getting hacked, it was like clockwork. And then one year, the most recent year, I think it was the fourth year of having a compromise that was costing them multiple millions of dollars. The attacker also stole systems around, or information around, four very proprietary systems. So it was highly elevated. I went in and was talking to them and said, okay, well here's the network environment, the topology, et cetera. And there was the component of Systems there's like 50 or 75 that were Linux and Unix systems. And I said, well, what have you done to look at persistence of the attacker in those networks? I understand what you've done in the Windows environment. They said, oh, no. Our incident response team told us this attacker does not target those types of systems system, so we don't have to worry about them. And I basically said, bullshit. They don't target those types of systems. They've got the capability to reside there. I sent Daniel and his team down and I think they found seven instances of compromised systems in the first two days.

Yeah, yeah, they were on the systems.

Yeah, they were, yeah. But they would do their response strategy and they would clean up the Windows environment. Then the attackers would slowly migrate back out of that Linux and Unix environment. I think you actually even found some dormant VMs that were windows, VMs that were on those Linux boxes that when reanimated, reintroduced the attacker back into the network. But they were making a decision that was costing them, I think it was 1.5 to $2 million every year, plus the exposure of these critical systems. Living with that risk because they had a bad piece of intel. And that bad piece of intel was, well, the attacker doesn't target these types of systems.

Well, there was also another relational dynamic with, I think the incident response company they were working with, which was, it was. It was kind of arrogant and really pricey. And some of their findings were, well, let's just re. IP number your, you know, 10,000 nodes or whatever, which you, you know, dealing with the technical folks on the, on the floor there, you know, you could just see their eyes roll, you know, like, what? We're doing this again? We're gonna, you know, like, yeah, just, you know, they weren't really building bridges, you know, so.

And they didn't have an endpoint for those Linux and Unix systems. Right. So there was no value in them. There was no upsell to the customer.

Right.

Whereas there was in the Windows environment where they kept signing them up for, well, run this engine, you know, detection. Like, you know, do all this this year after the incident and you know, we in there me identifying the kind of the threat intelligence issue. Daniel going in on the technical media investigation side and the client for, for multiple years and this had been an annual for them.

Yeah, it was. Yeah, there's a lot to see that.

In a lot of, a lot of different ways. I mean, we run across companies all the time that, you know, we don't have SIM coverage on all these machines because we can't afford it. But they, they report well, we have a SIM. Yeah but it's only catching 40% of your systems so I don't know if that's necessarily useful. I spoke to a company just yesterday. It was, they're very happy about their two factor solution. It's like, oh, you know, it's great. Two factor out all those systems. So what about your Linux boxes? We don't do it there. Oh, okay, cool. I've got some data on that. Let me, you know, let me share with you how to do that. Okay, great. I was like, what about your AWS environment? Because it was huge. Oh no, we don't do it there. That's, you know, you get locked out of that. And I was like I don't.

But somewhere on a risk management systems. Yeah, somewhere on a risk management spreadsheet somebody checked the box for two factor authentication.

Compliance officer. Got to go. Bloop. You know. Two factor. Yes, we have one.

I always like to say that our security aspirations, you know, that we demonstrate through checklists and compliance based approaches never match our security implementation.

Right. You know, and again, I think it's superhuman, not superhuman, but human in general to take those approaches because people look at it externally. This is hard. You know, we have to do all of this. Can't we just do the part that's easy to implement? You know like, it's like you could, but it's going to come back around.

I'll just put this out there. I mean one of the things that I'm kind of trying to focus on in Shadow Dragon and Packet Ninjas and anything where we're interfacing with people is let's try to push our message to be as concise and focused as possible because you know, I want to have the awareness that they only have so much appetite for X amount of information that we can throw at them. So how can we really focus on that particular skill which has nothing to do with the craft and it has everything to do with knowing how much can we feed to make it applicable and concise, you know, and that and then maybe follow up with some, you know, where are you in this maturity model type thing and just kind of walk with them because if you can walk with them for three or four years then there's going to be more success versus oh, we're just gonna, you know, have a one year, two year engagement. It's, these things are complex and your, your business is complex and your network is complex and it's organic as well, you know, so sure so let me.

Throw that one out there because I, I see this a lot, of course, someone dealing with breach situations, but there's a lot of people out there. You know, we're wearing that CISO CIO tag that are just woefully unqualified to be doing that. I'm just throwing that grenade into the conversation.

Well, Matt, I think you need to write a book on translating some of the technical things to policy. So there's like a framework for that, that message. Right, because that's a, that's a super hard thing, you know, like.

Yeah, now I'm working on it. You know, I've tried over the years, you know, the black cat. A couple years ago I did a session in the career track area called when hackers Briefed the board. How do you translate the technical aspects of cyber security? And part of that is also just understanding how to communicate with people. You can be the best cybersecurity person in the world, but if you don't understand how to communicate that up the chain, you're going to run into issues. So, yeah, that's constant. I am hopeful, you know, if I look at folks that took CISO positions over the past year, at least in my network, I see a trend towards more technical, kind of traditional computer security, cybersecurity folks taking those positions. But yeah, the other aspect of it is that as with any kind of rapidly emerging market, the requirements for staffing got ahead of the capability of the market. So there were a lot of folks that got put into those positions or kind of manage their career into a CISO position that really didn't have the experience or the skills to do a good job of it. Right. So you've also seen some getting fired and getting removed and you know, kind of some short lived careers. But I'm hopeful over the long term. I encountered the same issue when I was doing stuff with the military back in the 90s. Right. Where you had moving up the leadership chain. They didn't understand technology, they didn't want to deal with technology. And now if you go and talk with any of the folks in leadership positions, they get it. I mean, they are users of technology, they understand technology, they know what two factor authentication is. I have an infamous story from 20 years ago trying to get the head of Special Operations Command that changes one character password. So I'm hopeful that within the executive ranks of these major companies that we're going to go through that same evolution. Evolution over time.

Yeah, yeah, I like the hopefulness here.

Yeah, I prefer to be an Optimist. Yeah.

Yeah.

I'm not even necessarily convinced all the time that, that the most technical CISO or CIO is the win. I've seen some really good people that are almost more just, just, just high quality managers at that level that they're not incredibly technical, but they have their, here's the things we need to get done, here's the quality people I put into, you know, to work those things for me. And they have incredibly successful programs. Yep.

If you're good at building and managing teams, you can be successful. Right?

Right.

You gotta be relying on the experts that you hire. Right. If you can build a great team, you don't have to be the most technical ciso. You can be the one that manages the process and then manages it up the chain. Right. Some of the best CISOs are the ones that do that. As Daniel alluded to, they can translate the technical into the business context and that brings tremendous value. Right. They're the ones that get the budget. They're the ones that are actually trying to manage risk. They're the ones that educate the board. They're the ones that are pre preparing the board and the executive team for an incident, doing the tabletops and, you know, just the stuff that gets them conditioned to the world that we live in now.

Before we kind of move on, Matt, I want to be conscious of your, your schedule. If we can push farther, I'm okay talking a little bit more. If you're pressed for time, we can get to the.

I'm good to chat for a little bit longer.

Okay. So, Brian, the hand grenade topic past the chatter for today is actually disinformation. And you know, like, there's a lot of talk about this, there's a lot of new experts on this. There's not really that many new books on this craft from my perspective that, that I have in my library. So let's just kind of, I want to open the topic of, you know, what is disinformation? You, you guys are both in the D.C. area. You get to maybe hear a little bit more rhetoric than, than I do and maybe have more conversations in, in those realms and how does it impact information security and some of the things that people are talking about right now with election security and, and all that. You know, what's, what's real, what's not real is, is maybe even this whole topic and thread. Disinformation. You know what I mean?

You know, from my perspective, disinformation is not new. Right. But it's the means by which you can Propagate disinformation is, is changing as technology changes. But I think that less of it is really about disinformation with any sort of strategic outcome other than the outcome of us becoming less trusting of the institutions in society. So for me, disinformation is less about getting us to pursue a particular belief system or particular idea and more about getting us to argue with each other. And then when the outcome is not the outcome that I personally prefer, that I feel like I have a legitimate backstop for discrediting that. So that's one way that I think that it's transitioned over time is that it becomes really about targeting institutions of trust. Voting in election security is a great vehicle for that. Right. Because voting is a trusted institution of a democratic society and the intent is not to go in and hack and change the outcome of votes by hacking into voting machines. The intent is for citizens to question whether that is possible and that that happened so that when the person that they prefer to, you know, to win the election doesn't, that they can question the entire legitimacy of it or that they're, you know, we spend time arguing with each other on some of these, you know, distraction type topics. So for me that's kind of the big disinformation issue that we're dealing with is this diminishing, these attempts to diminish trust in institutions. And if you, if you push that out, you know, and I have a piece I did on OODA Loop a couple months ago that talked about, you know, what some of these state sponsored actors would do in the future. You start to think, well, what are other trusted institutions? Right. Financial institutions are trusted, you know, the other government type benefits entities, I mean entitlement type programs. There's all these different things that you could start to expand that, that sphere of influence and disinformation into.

Yeah, yeah, I, I kind of look at it too. I like, I like your focused view on that and that kind of like distills it down to something very actionable to anybody that wants to start looking into that. Kind of the way that I've kind of looked at it too is like you've got all this social media stuff out there and really it's, it's very polarizing. You know, it, it almost doesn't just become an echo chamber, it becomes a rage echo chamber, you know. Yeah, you're either binary yes or no. And you know, and, and I hate that because like, I just hate social media in general. I hate external self expression out on the media. Like it's Things are complicated. It's easier to have a complicated discussion over a cup of coffee or in, in front of each other than this, this drive by truthing of my way. And I'm also the worst of it. You know, I got off social media 10, 12 years ago because I was like, you know, after 10 o' clock I'm just, I don't want to know all this stuff, you know, like I don't want to go into rage mode if I disagree. And I think that the, the outset of everything is just in this rage mode on Twitter and Facebook and whatever the platform is. We just got to be okay with disagreeing, you know.

Yeah, no, absolutely. And I think part of the issue is right is nobody is going to run a disinformation campaign that is focused on bringing people together or finding middle ground. Right. So what you wind up with is the focus is on the extremes and the extreme drives, the radicalism on both sides of the equation or on, you know, the sides of an issue. So what you wind up with is that it ends up driving perspective to the extremes and kind of drowning out that middle ground because there's nothing that's, that's reinforcing it. So yeah, I mean I think that's a key point and that's one area where, and we've seen it with topics over the years where the more extreme they can be, you know, with the topic, the more the energy they get associated with some of these campaigns.

Yeah.

Because that's what people latch onto.

So we're talking about it around the edges of politics here. An actual example of where just bad information like and this is just recently, so it's probably three months ago, sat down with a ciso. One of the things we were talking to him about was, you know, blah blah, multi factor authentication. I feel like I just repeat that same five topics all the time to these folks. And he looked directly at me and was like oh well you know that stuff doesn't work. I was like what? He's like no, no, I read, read some articles, you know, blah blah that it multi factor, two factor, it doesn't work. And I was like I don't, I'm really not with you on this. And what he was looking at was I don't know if you guys recall the midnight had done a little presentation on sort of an edge case example of where he could beat some, some two factor system on some web based apps. And you know, but the, but the subsequent reporting of that done by non technical reporting types was oh two Factor authentication is dead. Two factor doesn't work, blah, blah. And, and the CISO had actually read those without ever reading the original, you know, presentation information. And it was that re, re. Re. Kind of dumbing down of the follow on reporting to what he had read was just oh, two factor's been breached, doesn't work, you know, blah, blah. It's all garbage. You know, don't, don't invest any money in this by, you know, reporters who didn't know what they were looking at and then re. Reporters of that information, you know.

Well, that's a, I mean that's, that's an example of pieces of information that just keep getting sliced and diced and if, if the root of it isn't true, the, you know, the, the first statement or the second reframing or the third reframing, you know, it's, it's going to get diluted, you know, like. One of the books that was really helpful for me in looking at the history of this was entitled Disinformation by Pasipa. I think that's how you name his last name. He had also wrote Red Horizons in the 80s. I don't know if you guys have read that book. It's amazing. One of the top KGB defectors that was in charge of some of these programs had come over to the US and heavily influenced the, you know, Reagan, Reagan's perspective on the Cold War and that kind of stuff. So one of the things that he really kind of illustrated was a manner in which you kind of start rewriting certain pieces of history to start changing that narrative towards that divide and conquer in the long run and, and being very skillful at that. And they even had numbers on, you know, how many people were employed in that craft in Russia in, during the Cold War. It was, you know, vastly more than any other program that they ever had. But it was the most successful kind of wrapping up like where I see that in everyday life or reading news or anything like that. Like there's a saying that, you know, it's, you know, hold every thought captive, you know, and you've probably heard that before. It's, you know, capture something, analyze it, you know, and I just try to ask questions like, you know, well, who wrote the article? What's the title of the article? Or is there any source in the article? Is there, you know, what are the true artifacts and then what are the things in a timeline? If I really wanted to geek out on this, that may sway truth one way or the other because somebody has an agenda, you Know it.

I mean, the problems that you're describing A. I invest a lot of my own personal finances and trying to help solve it with the ooda loop.com website. Right. The whole reason that that site exists is because we wanted an environment where you could have practitioner based, you know, pieces on these topics that was not done by journalists. Right. With journalists you have this drive of a. They might not understand the topic. They're dealing with a huge portfolio of subjects. Second, they have to be sensationalist because their economic model is based on driving eyeballs and views to it or in our industry. The other problem that we have is that the practitioners that were writing were doing it on a kind of pay to play type basis. Right?

Right.

Information was biased because they're economically incentivized to go and write a particular article. I think there is just a real requirement for an ecosystem within our niche. That's why we did it with the OODA loop site for. Okay, what is the subjective practitioner based, expert based view on this topic that decision makers can turn to when they have a curiosity but at the same time also building a network. If the answer is not known to you, who is a trusted resource that you can connect to? Do we also cover the open sources? Right. You know, we do a. In, in addition to the original analysis, we we distill down, you know, what are the top five cyber security articles you should be focused on global risk articles, technology articles. But there is definitely value, I think to trying to foster an ecosystem that is not journalist derived. I love journalists. They've got a role, but they can't expect them to have expertise in all areas. Nor can you expect them to ignore the economic realities of the industry that they're in. And driving traffic and is not pay to play. I mean, the pay to play stuff is probably the most frustrating for me because that is often the originating source for some of this stuff that distills down into inaccurate things that are deemed by a fact at the CISO level great.

And defining pay to play is sometimes hard when there's not necessarily just C Span anymore. There's a lot of conglomerates that are making money on ads and eyeballs on the screen and, you know, contracts for cable news or whatnot. And so it really makes it hard to discern from, you know, mom and dad's perspective at home if they're not taking apart every single layer.

You know, it's hard for sophisticated folks in this space oftentimes to discern that pay to play element. Right. You got to kind of Dig didn't look at the economics and the sourcing and the kind of, why does this piece exist? You know, and it requires effort. Right. It's kind of that you got to do some thinking for yourself.

Yeah. So many of your social media platforms are echo chambers of what your belief is anyway, Right. You know, it reflects back to you what you already want to show you, you know, so.

Because the algorithms tell it to, right?

Exactly. I mean, but that makes sense. It really dangerous to your average person. Right. That doesn't realize what they're seeing. And, you know, the world agrees with me. The algorithm just pointed you to the people that greets, you know.

Yeah. And it's very dangerous for kind of group think dynamics to have ecosystems like that that don't expose you to alternative points of view. Right. I mean, it's classic group dynamic studies that if you have 10 people in a room and one person is really radicalized and all that you're exposed to is that one person speaking, you're likely to get a majority of the room to follow up. If you get one dissenting voice in the room, one person that asks a tough question, then you start to split the room in half. I think part of the problem with the social media environment is we have created these echo chambers where all that we're seeing is this stuff that is radically reinforcing previous ideas. So you don't get that critical question, you don't get that alternative perspective that might sway you. There are some people that they're just, their, their position is their position and they're not going to be dissuaded. But I think a bulk of the folks, you know, at least the people that I interact with, they're looking for sources of information that help inform that judgment. If all they're ever provided is one perspective, that becomes a very dangerous environment.

But that's, that's maybe part of, the hard part of this whole situation is, you know, I can go find articles to support almost any question. Crazy idea I have. Right. No matter how wrong it is. I've got seven articles here that prove my point. Not exactly, but it just exacerbates the problem because they can do that. Right. Always find some publication somewhere.

You can. The issue becomes, now, if you have people that aren't seeking out that information, what are the algorithms delivering to them? Right. Is it that I have. 60% of my friends are conservatives. I'm getting that angle. Is it. You know, I think that's part of the danger around some of the, the, the dark science, around getting Attention and what shows up in your feed and what doesn't show up in your feed. My favorite iteration of social media environments is the pure timeline based, like I just want to go and scroll through everything. I don't want an algorithm to distill it for me as to, here's what you should see, here's what we think you should see. And almost every social media environment has moved to that model. We're going to pre process for you and give you stuff that we think has the highest probability for engagement. And oh, by the way, engagement is also tied to this advertising component. We're going to show you content that best suits the advertisements are going to display in frame as well. And that's just within our society. Right. I mean, the problem gets a lot more exacerbated when you move into some of these cultures where they don't have a cultural conception of a Photoshopped image. Right. We can look at an image and I can be like, oh, hey, you know, Arnold Schwarzenegger was never in an episode of the Office. That's pretty funny. But you have some of these cultures where the idea of Photoshop, the idea of imagery being false, does not exist. And these platforms are being used to be hugely influential in those environments because anybody with a rudimentary technology skills can. Can now change truth. And it's a cultural predisposition to just accept what's in the image.

Yeah, yeah, yeah.

It's.

It's a hard, it's a hard space right now. I mean, especially with there's there. I mean, kind of going back to the institutions that are getting attacked right now. We've had the elections, we've had freedom of speech, you know, the Constitution, you know.

Yeah.

You've got freedom of press or the, the reliability of press, you know. So is this state sponsored or is this the unintended consequences of everybody being on social media all at one time and bad algorithms?

Yeah.

I don't get a combination of the two.

Yeah, it could be a combination of two or three, you know, like, we just don't know. So I'm gonna put a period on that because we're still.

You could go down that rabbit hole for days.

Yeah. But I think we could follow up on it in the future, you know, like.

Yeah, that'd be great.

I've got a lot of things that I've been, you know, we may want to pick a topic and then we all come to that topic with what's our timeline?

Sure.

And see how, you know, maybe an example for people to Say, hey, you know, these guys may not have all the same perspectives, but what's, what are the real artifacts that they pulled up and what's the outcome for that? And yeah, you know, we're not all going to agree.

No, Yeah, I would hope not.

Yeah. So the last thing we've got, our inbox, which is a part of our Q A. One of the, the common things that we get a lot is why don't we have a big crap map? We call it the crap map internally Shadow Dragon. So we can do location and geofencing and more sentiment analysis, which as you guys probably know, that doesn't really always work. You know, I haven't seen it work yet. Have you seen it work in idefense or eyesight or anything else like truly work other than a good presenting tool?

Yeah, I mean the, the Pew, Pew maps, you know, or management like to see them, you know, that's, that's the reality. They're, they're, they're interesting. There is useful data, right? Is being able to associate at least, you know, a geographical presence with something, even knowing that that could be spoofed gives you, you know, some insight, right. At least lets you see where stuff is routing through. The problem that you have though is being able to distinguish that kind of informational view from an intelligence view. We know how difficult attribution is, we know how easy it is for the attacker to spoof the geolocation of where they're originating from. So I don't mind the map so much for the entertainment value or giving you nice imagery. What I worry about is when somebody tries to derive information from it and they're like, oh well, wait a minute, we're being attacked by people in London. Well, maybe not necessarily. Yeah, wouldn't make any judgments on that. Let's block all UK based IP addresses. So that's good. It gets a little bit more interesting on the attribution side around mobile devices etc. I'm always amazed that people don't understand the extent to which your devices are tracking you. And it's not just the manufacturer, it's the advertiser id and most of the applications that you're dealing with. Geo fencing maybe becomes interesting in that context because if I'm on Twitter with a mobile phone that says Daniel must die and it shows my geolocation is outside your house, from a protection perspective, maybe you want to have that information. I don't think there's a simple answer. I think there's a combination of different values that you can Go into with eyes wide open looking at a topic like that.

Yeah, I agree. I mean, we've kind of taken a stance of like making a workflow that can, can extract things if they're there, but also, you know, forcing people to think and plot, you know.

Yeah.

So like if they do want to put something into a visual form and have that, you know, visual spatial analysis type platform, it's helpful for that. And some things can be checked as verified or not verified, you know.

Yeah, yeah. And you know, the verification piece is the difficult part.

Yeah, yeah. And then on sentiment analysis, I mean, that's just like such a huge can of worms. Like everybody asks for that all the time and that's kind of an observation.

Yeah, I mean there, there is a science to it. But I feel like as we talked about the problems with variables around machine learning and AI, that it is a science that is subject to high level manipulation. So very difficult to kind of discern true sentiment. Vice manipulated sentiment would be my concern.

Yeah, that's my concern too. I mean like we looked at some of the phrases and idioms used for some of the folks that are kind of moving through the firewall of China to get around that and stuff. And that's changing on sometimes an hourly to daily basis.

Sure.

Which you can't really even. You can't put really a learning, you know, stem on that really to be actionable for anything that's worthwhile, you know, for a corpus of data. So in the that area or in language, language sets that you're just. Different regions are going to have a lot of different things that.

And language is evolving too.

Right, right.

I've learned that with my kids that the, the phraseology and meaning that I used when I was a teenager is different than the way they use it today.

Yeah, definitely.

So you've got to have, you know, almost kind of adaptive dictionaries as well to make sure you're getting the true context or the ability to differentiate that. I just use a particular term and my intent was different than the 17 year old that just used the same exact term.

Yeah, I agree.

Then you take into account multiple languages and cultures, it gets all out of hand.

Absolutely. And an increasing dependency on machine language processing. Right. Which means that you're getting the same interpretation based on algorithms across that as well.

Yeah. So beware buyers. Here's to you, AI.

Beware.

Well guys, we've gone a little bit over, but I think we're pretty much on time based on when we started. And I think that if, you know, you guys want to have another revisit on disinformation and what we all research at some later time? I think some. Some of our folks out there, all three viewers, might like that, you know?

Absolutely. No, count me in. You know, I'm always down for good conversation, so.