EP117 Can a Small Team Adopt an Engineering-Centric Approach to Cybersecurity?

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit your subscribe button. You can follow the show and argue with us on Twitter as well, twitter.com/cloudsecpodcast. Additionally, our LinkedIn page has been pretty active lately, and we've had a lot of fun engagement there. So if you want to come join the conversation, check us out on LinkedIn. Anton, we are talking about engineering-led security today. What does that mean?

Well, that's exactly the point. It's kind of an interesting mystery, and as we explore it during this podcast, we also explore the resistance to it. So it's interesting how it's for almost every concept in security, there would be a group of people who resist that.

Yes. Security people are good at resisting. Clearly, the Borg never met security engineers. My question is, if it's not engineering leading security, who is? What is this in contrast to?

Okay. But this is actually very simple. And we had a bit of an argument with a guest about it. My initial impression is that if you are not engineering security, Your approach is to consume security. I want to go to a store, bring my wallet, whip out my wallet, find the biggest, best vendor, and I give them the money, and they give me this big package that says security, and I get it, and I consume it. That's the opposite of engineering-led security.

But who's the person with the credit card if not somebody from engineering? Where are they from?

The person with the credit card, well, on the customer side is the customer leader. It may be the security manager, but they don't want to engineer anything. They want to buy and consume security. They're not thinking of engineering security. They're thinking of consuming security. Buy anti-X, never suffer from X. There's no engineering involved in their mind.

Oh, I see. It almost didn't occur to me that somebody might think that way. Maybe listeners, with that revelation from me, let's turn things over to today's guest.

Hello. And today's guest is Maxime Lamotte-Bressard, founder at Lima Charlie. Welcome to the podcast.

Hey, super happy to be here.

I wanted to— we, I guess Tim is here virtually— wanted to explore the whole topic area of engineering-centric approach to security. And you sort of, when you built Lima Charlie, presumably you followed that. So why don't you describe it to us? What do you mean by engineering-centric approach to security?

Yeah, engineering's approach to security, I think it's sort of about the philosophy really behind it. So it doesn't necessarily mean Hey, everything is code, right? So it's not necessarily, you know, the software engineering approach to security. I think another kind of way to view it is almost to think about it like, you know, the scientific approach to security.

Mm-hmm.

So to me, there's really sort of 3 big poles to that. So the first one would be to have the ability to reason about the security posture, right? I'm kind of framing this from the perspective of like, I'm in an organization, and I have to do security. So one would be reasoning about the security posture. So what that means is—

So you want security to be reasonable? That pun is in memory of Tim, who is not here, but who will listen to it eventually.

Both would be good. But reasoning is really the idea that, you know, if I put it as, like, a sentence or a situation, right? If I'm the CISO in a company, or I'm, you know, working in security in a company, And the CISO comes to me and says, hey, this, you know, thing X just came out. What's the implication? Are we safe for that? And if the answer is, well, let me talk to the vendor and I'll tell you what they say, that's not reasoning about cybersecurity. For me, reasoning about cybersecurity means to have the ability to go and say, okay, here's what this, you know, malware actor vulnerability, whatever it is, means here's the potential impact on what we do. Here's how we're protecting against it, or if we're not, how we could detect it. And if that happens, how we would react, right? So it's sort of about having the blueprint for, hey, exactly what's going on here to be able to analyze that security posture. The second one would be to test, to be able to test that security posture. I think that's fairly straightforward to explain. And the last one is to have the ability to impact change on that security posture, right? So—

Oh.

If I put it to that same scenario, if the answer is, well, actually, we don't, you know, we can't detect that, we don't know if it's going to happen, and if then we have to say, well, let's put in a ticket at the vendor and wait until they come up with something, that is not being able to impact change. And being able to impact change is, again, to pick a real-life scenario, right? WannaCry happens, it turns out to be the easiest thing in the world to see and stop. You know, it's a static executable name. It doesn't get easier than that. So having the ability to impact change means I don't have to wait to see, you know, whether my vendor's ML, you know, detects and prevents it. I can actually go and have governance. I've got the keys. I can do the thing.

So to kind of go back to my kind of former analyst brain, so then engineering-centric and your pillars would be kind of the opposite of opinion or gut-based? Like, what is the— like, how many people practice security, if the CIO or the CEO comes to that security leader and says, are we secure? They say, yep, we're good. But it would be based on opinion and gut rather than on data or evidence and discoverability that you described, right?

That's correct. And I would add to those terms, promise, right? I think a lot of security today is very much promise-based security. Which is— I'm not saying it's a bad thing, but, you know, I think it's sort of important to identify that, you know, if I buy vendor X, I'm really buying the promise that they are the best in the world at doing this thing.

Yes, that actually does make sense. And to me, promise isn't necessarily bad, but it also isn't necessarily reliable because, as my classic example from my Gartner days, if somebody outsources security to an MSSP and they really screw up very badly, and you lose your business, they lost a client, but you lost everything.

Mm-hmm.

So sometimes the promise-based just means that it's just tricky to substantiate and to prove and to reason about it, as you pointed out.

Okay.

But you didn't use the word, these words. You kind of said engineering-centric or engineering-led. And some people hear this and say, engineering, I got to code. I don't want to code. This question, I almost immediately thought, I got to ask Maksim that question because I see people who want to consume security rather than engineer it. So for them, almost promise-based is kind of almost like a positive. If they see a well-known vendor name written on a piece of paper, they think, wow, this is awesome. This is secured by vendor C or vendor P or vendor— let's pick a letter that doesn't map to a well-known vendor—

X.

And then that's good enough. So what do we tell people who just counter and say, we want to consume security, we don't want to engineer it?

I think that is absolutely correct, right? Not everybody can, you know, have, you know, malware ninjas and really, really, you know, advanced and mature security teams and all that. So I think that's definitely not, you know, where my mind goes. The way that I would kind of compare it, I would map this to the action or the requirement or the need for a website, Okay. And so, if I'm a tiny company, right, I run like a cleaning company somewhere, and I need a website for my enterprise. What I'm going to do is likely, let's say, you know, go on Squarespace and get Squarespace to do that together, right? I'm not an engineer. I just need the easy button. To me, that's like, you know, using this promise-based, you know, I'm gonna go, or maybe I'll go see a small local company so they can build me, The last thing I'm going to do is I'm not going to go to AWS, right? If I'm a tiny cleaning company, like, I can't go and actually build my website on AWS. However, if you're General Motors and you're trying to build your website, it's probably really more than one website. It probably integrates with supply chain with a lot, right? And for them, they're not going to go to Squarespace. they are going to go to AWS. So, I guess what I'm diluting this to is that there's different types of organizations, and that is absolutely fine. But what's interesting is that Squarespace is built on AWS, right? So, there's like, there's a chain of value there. And that's how I think about security and AV companies and MSSPs and all of that.

So, basically, that means it's almost like you're describing the world where it's either engineering-led security or engineering-led security under the hood, or you have not much security, or you don't have security you can rely on. This is like, maybe this is pushing it all the way to cranking it all the way to 11. But kind of, that's what I'm hearing.

That's right. That's right. I think it's sort of at a really macro global level, engineering-based security is sort of the utopia, because utopia tends to be not a place where we can be. But it's sort of the general goal, right, where we're heading. So that as an industry, again, globally, right, we do things, you know, in a more reproducible way, in a more defensible way, right? I mean, you know how difficult it's been historically to sort of defend budgets and products and all that.

So that does make sense. And I think that also allows me to actually use this argument to people who say, you're suggesting we become more like Google and rely on engineers for a lot of tasks. We don't have the engineers. And the answer is, okay, actually, then you would consume a service from somebody who does have the engineers. It may be us, it may be somebody else, but ultimately, you will benefit from engineering-led security, even if you are not hiring engineers to build it. Is that fair?

That is exactly it.

Yeah.

Awesome. Love it. Okay. So we can almost stop the podcast now because there are a few areas I wanted to touch. And when I was thinking together, we're thinking about the questions, The whole point of evidence came up, or provable security, and my mind went to kind of evidence-based medicine, when even though medicine is based on science, you still want a particular treatment to be tested based on some kind of principles. We don't really have that in security though. I couldn't think of a single example where security, except for maybe some kind of mainframe system design from the '80s, the rainbow books, whatever. Yes, I am that old. Is engineering-centric the same as evidence-centric? How do we make security more evidence-centric? I don't know. I'm giving you a bit of a loose question, but I think that—

No, those are— I mean, yeah, those are the best conversations, right? You know, I think for me, evidence-based security, here's my interpretation, and it is embodied pretty well from something like Atomic Red Team.

Okay.

To me, you know, it's the testability aspect of it. And kind of coupling this testability concept with a DevOps concept, which is infrastructure as code, right? So, to be able to think of your security posture as code or as a very structured thing, and then to be able to test it. And then, you know, DevOps really kind of starts to seep into that conversation where, you know, we're talking about CI/CD pipelines and All that great stuff that's come out of that movement. So for me, that's my interpretation.

That does make sense. And to me, this also points people who are kind of phobic of code, codophobic, I don't know, is there such a term? I think we're making it. Who hear engineering-centric and they thought, ah, we have to program. But you say, actually, well, yeah, you kind of have to program or somebody has to program. But my point is that you don't have to program. My point is that you would use evidence-based system principles and approaches, not that you will have to program, even though ultimately somebody will have to program.

That's right. You know, if you look at it from the DevOps kind of lens, that's exactly what you were saying, right? Like, will you have to write code? Well, you don't necessarily have to write it. But at some point, probably somewhere, I think there's a very meaty middle in all of this, right? Where we're saying, look, it's not necessarily about programming as much as using these principles, right? It's kind of like Terraform, right? Look, if you're building and organizing Terraform files, you're not necessarily coding, but it's sort of the critical aspect of it.

Okay. So, I'm happy you didn't use the 1990s example of if you're writing a shell script, you're not coding, but you went for a more modern one, and I'm grateful for that. But how do you do that? So, I think we established that it's an awesome idea, and I think we established that it has pillars. So, in practical terms, if you have an organization, large or small, let's sort of avoid this, like how large or small, rich or poor, kind of a separate story. What does it take to adopt an engineering-centric approach for an organization? And what would be better? What would it enable?

I think that's a great question. And I don't have a good, like, answer to that in a sense that I think it's really hard right now to do that. Because most of the vendor space is in the promise-based kind of region, step number one, you can't quite get there. So, I think people that do want to go there end up having to go towards open source, right? Because by definition, open source. So, now, there's a lot of drawbacks to open source, right? It tends to be kind of difficult to build or scale and all that stuff. But it does allow you to start using things like, you know, the Sigma project. Yes. Great thing. It's sort of, for me, the equivalent of the Atomic Red Team, but around detection engineering. And so, by using those, you can start to go in that direction. But as I said, it's really, really tricky. And I mean, you know, I won't kind of go into that big tangent, but that's exactly why we built Lima Charlie specifically kind of in that direction, right? Because we are essentially trying to bridge this idea of having a vendor, somebody that does take care of all the difficult bits and the scaling and all that stuff, but where you still have that visibility. And to me, in my mind, you know, if we kind of map that back to IT, what it means is the concept of a cloud provider, right? Somebody like a Google Cloud Platform, where you can have access to all of the things in Google Cloud in a very transparent way, and you're able to build that.

Yep, that actually makes sense. And I like this argument, because it sort of like reminds me of what our CISO Phil Dannebels highlights in his megatrends, He sometimes says cloud is more simple, and people freak out and say, no, cloud is hard. But actually, his argument is exactly what you just said. Like, no, actually, you can call APIs, you get predictable results, you have well-documented things, and the universe of these things is limited. You don't have to think which model of hardware power supply I use in my servers. Like, that's all abstracted. So it's kind of simpler and cleaner with cloud.

I think that's correct. And that's really, really critical in that It goes back a little bit to earlier in the discussion, but the idea is cloud can be complex for some people, but they're probably not really, you know, the people that should be interacting directly with that cloud. But at the end of the day, we've discovered in IT and in tech that difficult things exist, and they're very, very, very, very, very rarely silver bullets for them.

No matter what the promise is, right?

Exactly. Exactly. So, it's about harnessing the complexity, but in a fair And to me, that's what the promise of cloud providers, you know.

Okay. Dear listeners, I promise I didn't plant the point about cloud, but it's also kind of relatively clear that it is better in the cloud. It's almost like it's not a joke and it's not a planted marketing message. These things genuinely are better and cleaner and they are more secure in the cloud. So, as a quick side tangent, I was on a quest to find CISOs or security leaders who prefer cloud for security reasons. And it was a very interesting quest because these people absolutely exist. But if you go talk to sort of like a random audience of security professionals at the conference, they would deny their existence. They would say, no, what do you mean? No, cloud was pushed to us by business. We hate it. But it's a very— maybe it's an old stance. It's not the reality today. Cloud really is better. For this.

Yeah, absolutely. And again, right, it's complex problems. And I'll stop myself there. It's a whole super interesting tangent.

It's a super interesting tangent. So, let's talk a bit more about the practice. And I think that we kind of thought about how would you practice engineering-centric engineering, that approach. And I think that we sort of answered how do you practice with non-engineers. You rely on somebody who practice it, and then you sort of see transparently how they practice it. But what is the approach of doing it with a small team, with limited resources, when you're not hiring 10 people who are both security experts and software engineers? How would you approach that?

I think the best approach for that is to, you know, what's the expression? Eat the elephant one bite at a time, right?

Yeah.

So to pick a specific aspect of the security posture, whether it's, you know, endpoint security or, you know, network, or even go down to DNS, you know, things like that.

Mm-hmm.

And to start kind of applying those sort of 3 concepts that I referred to, right? The first one being, okay, how do we know— let's pick DNS, right? How do we know what can go out through DNS? Do we have the ability to go and reason around, you know, the rules probably that are filtering the DNS and the paths that DNS is taking? Then, the second one would be, okay, can we actually have a way to test that in our environment? So that we can prove that, hey, we have this mechanism to, you know, block bad DNS for, you know, XYZ. And yes, it actually works and it's there. And then the third one being, okay, how can we have impact if tomorrow, you know, we see on Twitter that, you know, WannaCry goes to 3322.org and we decide that we want to start blocking it? What is our process to go and be able to apply that? And I think by taking these small chunks and growing them over time is how you get there.

Hmm.

Actually, that does make sense. And the whole elephant eating argument to me applies in full force because I've met enough people who get overwhelmed right away when they hear about this. When we talk to people about autonomous security operations, our kind of engineering-led approach to SOC design, they freak out. They think engineering, coding, they don't have it, they lose it. So to me, the whole elephant eating and element-by-element thing and transparency These are good elements to practice.

And some are easier than others, too.

Yeah, for sure. But absolutely the case, yes. So let's try to kind of start combining that theme. So it does sound like many elements of software engineering would be borrowed as kind of security would remind software engineering a lot more. So when we tried to explain it to somebody about a SOC, we said, well, think about the DevOps. It's Dev plus Ops. In a SOC, you have content designers who write rules for your SIEM, and you have the ops people watching the screens. Why don't you combine them? You have the analyst/engineer who would do the detection/response tasks. So to me, this is a mental model we use for this. But you seem to apply it to more elements of security, and that seemed to be kind of interesting. How do you— almost like, how do you hybridize? How do you merge software engineering and security?

Yeah.

That's a weird way to think about it, but like, okay. So if you can say something awesome, it would be good. For sure.

Yeah, I think that the first statement sort of to inject in there is I would take out of the equation AppSec, right? AppSec exists, secure practices around development of apps. I'm not taking it out as in saying it's not good or we don't need to do it or anything like that, but I think it's different enough and well-defined enough that it's not really part of that picture. So, For me, what it really comes down to, you know, I think you said DevOps. It's a lot of these concepts of DevOps and with the idea of adding the analysts performing the security as kind of integrated into that picture. Because DevOps is what allows you to raise the floor of your security constantly, right? I think that's kind of the— one of the underlying things we haven't mentioned yet is, Automation, right, is critical so that your analysts— it's not that your analysts necessarily get better, but have less and less to do of the things that they know about. I used to kind of talk about that as examples of using YARA scanning. It's a pretty niche example. I fully recognize that. But I think it demonstrated where, you know, in a lot of organizations, there's some capability to do YARA scanning. And that's great. That's a great capability. And you talk to the security team and how's this being done? Well, somebody like our team lead has a reminder in his calendar every 2 weeks to take some of the YARA signatures that are kind of relevant and go and scan for that. And that's, I mean, that's good. That's better than nothing, but that is not automation, right?

That's very artisan. That's not industrial or engineering-led. That's very get a shovel and dig, not use it, not buy or build an excavator.

Kind of thing. Exactly. Exactly. So, being able to kind of just throw all of your signatures in one pile where I know it's being scanned, like, you don't have to think about it. It's automated. It's through infrastructure as code. It's tested. That is when you remove so many of those tasks from your analysts. So, that's kind of the functional bridge between the analysts and the builders and the DevOps, right? It's to offload a lot of those activities.

Okay. No, that actually does make sense. And I think that it's almost like there are some kind of threads in software engineering you have to pull and almost like then kind of introduce them in the security slowly. And then you build engineering-led security by almost like stealing ideas from software engineering one by one and by implanting them into the body of the security operations or other security disciplines.

Absolutely. I think software engineering and kind of general tech is maybe a decade ahead of security. Now, I'm not saying that in a bad way or like, you know, I mean, I'm part of security, but sort of from having gone from the very opaque development and setup of infrastructure, right? Like, what is the expression? Snowflake servers and all that, all the way through Chef and Salt and Terraform. And, you know, so I think it's really important to get a lot of those lessons. Absolutely.

Yes. And that's actually a superb conclusion because to me, the 10-year time lag is is real. It's like, who am I kidding? Yes, I think that in many domains, the IT maturity is 10 years ahead of security maturity. That's a fact. It's kind of like, we can cry about it, but like, it wouldn't change from just us crying. We have to introduce more ideas from that side. And again, at Google, it will be SRE ideas, and the outside world would be DevOps ideas. And it's clear what to do and how to approach it, but it just needs to be done.

That's right. And that change is going to happen from every direction in security, right? It's not up to vendors to push that or up to users to push that. It's, you know, the temperature is going to go up over time, and that's how we're going to get to the better place.

Yes, this is awesome. And this is a very optimistic view, which is, I guess, allowed in security conversations sometimes. So traditional closing questions, usually Tim does it, so I'll try to do it. We usually ask our guests for one tip on how to make this real, for example, how to How to make it happen and one or more resources, reading materials. And of course, it's fine to say Lima Charlie blog or something. We wouldn't hold it against you. So, one tip and some resources for the reading.

One tip, you know, I think we touched a little bit on it. I would say it feels cheesy even saying that, but to kind of really call out specific time and resources to start eating that elephant one bite at a time.

Yes, yes.

You know, we're all fighting fires all the time. So, it's really easy for that to go to the side. But sometimes you need to build up the fire department so that they have to fight less fires and, you know, things get better.

Yes.

In terms of resources, I mean, yeah, you called it. Definitely Lima Charlie. I think that's exactly the direction that we're going. So, we're always happy to talk to folks. But there are, you know, 2 other projects really stick out in my mind that we mentioned, but the Sigma Project, because it brings very clear transparency to the detection engineering. It's a kind of a great community and resource to to start going and experimenting there. The second one would be Atomic Red Team, again, around testability. I'm sure there's a bunch of other vendors that I'm not even thinking about that sort of are in the testability space and that kind of stuff. But yeah, to me, they're the best resources.

Perfect. Thank you very much for being on the show. This was very insightful, and I think that the audience can start applying some of this in their organizations without fear.

That's right.

Without fear that they have to code.

That's right. My pleasure.

And now we are at time. Thank you very much for listening and, of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter, @anton_chuvakin and @anton_chuvakin. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.