EP194 Deep Dive into ADR - Application Detection and Response

Secure running. Hi there.

Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior PM for Risk Analytics and Cloud Threat in Google SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of the Cloud Security Podcast listeners on our Google Cloud Security Community page. Without any further ado, let's turn things over to today's guest.

And today we have Daniel Schechter, co-founder and CEO at Miga Security, and the topic is, of course, EDR. Welcome to the show, Daniel.

Thank you so much.

So, let's talk about our subject. Given all the acronyms we have in the industry, we have, of course, EDR, which is now more than 10 years old. We have CDR that's been emerging, and suddenly ADR shows up. So, my first question is really 2 questions: why do we need it, and what is it?

In the simplest terms that exist, ADR is about detecting and responding to application-layer threats. You know, I think that the modern application landscape changed dramatically over the last few years with the transition to cloud, which created for many, many people highly distributed, complex environments with constant changes, not only introducing technical complexities, but also a lot of organizational complexities.

Mm-hmm.

You know, most teams today, when I'm trying with them, are really struggling because they're either trying to prevent vulnerabilities from hitting production Or they try to utilize their current detection response tool like the one you mentioned.

Yes.

In order to detect anomalies on the server or initial request of the application itself. But the problem, I think, is again, in modern distributed application and this velocity of change, both approaches are not really sustainable because AppSec team ability to really review and understand each and every change impact on the code is really, really, really hard, if even possible. And the current tooling ability from an anomalous behavior standpoint to identify if, you know, if something that changed in the application, is it indicating an attack or just, you know, there was a PR 5 minutes ago and I didn't even know about it. It's also, you know, making things a bit harder. And so you need a tool that is designed to thinking across this complex environment for you, your clusters, your containers, your business logic, First party, third party, essentially to understand how requests look like. And you need something that allows you to proactively broadcast the risk and monitor it automatically. As you mentioned, same as EDR kind of, I think, changed the way we're thinking about endpoint security over a decade ago. This is kind of where— why we were ought to try and bring this approach to the application layer.

So if you've seen from the draft questions that we send you, you notice that there's a little bit of a skepticism, kind of like a tiny bit of skepticism coming through, and I think that you would— I asked you a question about the SIEM, but so far, funny enough, what you described is actually fairly convincing. And I'm sorry for my audience who expected me to kind of rage about how you're wrong, but I don't think you're wrong. And so, let me ask you to test this hypothesis from one side. Most of the applications you're describing are modern applications. That means they're probably in the cloud. You're not talking about something written by the Germans in the 1980s.

True.

Why isn't it just a feature of CDR? Because if these modern apps are in the cloud, surely people who are trying to build cloud detection response platform would care about this. And of course, I suspect that EDR providers, many of whom who are trying to show up in the cloud, they may try to do this as well. So why isn't it a subset of CDR? And what is the key differentiator? What is the magic that sets it apart from People trying to do EDR or CDR, and what does it change? Sorry, lots of questions, but you get the theme.

It's a good question, you know, and it's a question that when we started this, of course, we asked ourselves as well, right? I think CDR was already a thing. EDR, of course, is a thing for many years. And I think that there is essentially a big differentiation that actually allows us to play very nicely together. And I think we need to think about how it works or what are we trying to solve, each and every one of us. I think that For CDR, it's all about the ability to understand, is my cloud environment, an ability to ingest the data from my cloud environment and understand how it works, is in line with how I want my cloud environment to look like? When you're thinking about applications, I think even from the sources, and again, going back to cloud environment, so you're taking a lot of information from the cloud itself, and you're taking a lot of information from the network level and from the OS level. and multiple things like this. But when you're thinking about applications, applications today are going beyond your cloud environment. An application today, flow, or combining first-party and third-party business logic tools distributed across subprocessors and, you know, in your own environment and across multiple clouds. And the type of data that you need in order to understand it is also a bit different because it's not only to understand your OS level or network level, you actually need to understand the business logic itself. Because if you had a request coming in that actually triggered a subprocessor outside of your cloud, which generated another input to your application within your cloud, that's run and exec within, again, within maybe— it's going beyond the areas of your initial cloud. And at least for us, this is why we say, because this focus on business logic, and it's not only business logic, but essentially it's like an application-level logic.

Yep.

is so unique. You must have, like, a tool which is designed and built to understand this. It's the same you had the tool to understand how the cloud actually works.

A business logic and application argument is a really strong argument. So, I would definitely buy it from that point of view. But the nervousness that gives me is that in other areas, when people try to, say, deploy an application scanning tool, even before that, they kind of said, No, no, no, we are not just doing a vulnerability scan. We understand the application logic. This is kind of hard to do. And I mean, you're a startup founder. Hard to do is what you do. But it is particularly hard to do.

Yes. And I think it's really hard to do. And I think this is why when we thought about it, we said, it's a big enough problem to put the time and efforts to solve it. And I think that over the last few years, there actually was a major change in the technological world that allows it. With the introduction of concepts like distributed tracing, which were not always, I would say, adopted by maybe the security side of things, because again, you need to think about an application and then correlate it back to the broader environment, to your clusters and containers and to your WAF tools and your SaaS tools and all of this. But if you're focused on distributed tracing as a core, you're actually able to start understanding how code execution flow looks like. And for us, at least, Applications are a set of code execution flows. And if you are able to identify how a request essentially looks like across multiple components in your application, you are essentially doing again what EDR did in the OS level. You can actually do it in the application level. And this is why we believe that this is on the contrary. We are, as time goes by, we want to integrate with many CDRs. complement each other because we are not focusing on, you know, detecting anomalies in your infrastructure level. We're all about understanding how if your code execution flow doesn't really make sense anymore.

Yes. And I think that it's the division to me is somewhat visible between the classic, I forgot my S3 bucket and it's open, sort of CSPM, and then CDR, it kind of would be the abuse of that or people trying to move data. But then here you're looking at an application and what's wrong with inside the application. So, this is kind of— this part to me makes sense. And I think I'm almost embarrassed to ask my obligatory SIEM question because you've sort of already outlined why the answer is clear, but I still want to try. And the reason for this is even in the EDR space, you have an occasional debate about people who say, I'm just going to get Sysmon. I'm going to get Sysmon into some kind of a non-charging-per-dollar SIEM. Let's not go into details about which it may be. And then, I'm going to replicate most EDR functionality. And then a bunch of people show up and say, no, you're an idiot, it's not true, here are the good reasons. So, I think that I'm going to try to make it a bit more complicated. So, imagine that you're going to send all the application logs and possibly eBPF traces into the SIEM. And while context would be missing in a traditional SIEM, I mean, there are pretty modern SIEMs that try to stitch a lot of context together with logs and they kind of like dereference IDs and bring things and link tables to. So like, SIEM is— this isn't like 2007 SIEM. So what is your answer to why can't I just send my application data, including, say, eBPF traces, to a SIEM and achieve the goals of ADR that way?

So it's a good question. And I think, like you're saying, there is a reason why this debate is actually happening in other layers of the detection response suite. I think that application, going back to what I mentioned before, I think that The ability to actually correlate multiple points across the same request is probably the hardest part about doing detection response in applications, because it's not about the singular event sometimes that is problematic, but it's how, you know, an unusual flow created the singular event. And I think that once you are understanding this and the other part, which I'll talk in a second, we actually adopted like multiple other detection response tools.

Mm-hmm.

We actually adopted a very, very strong attack scenario analysis, which is happening all the time. And by having this ability to have this bank of always step ahead attack scenario base and always think about I'm managing my application, I truly understand how it works. Now I can tell you if what I'm seeing is either telling me that an application attack can be happening or is happening is much more than the ability to understand a singular event in time. The amount of context you need to have there is so big, and the amount of deep understanding of how the application works you need to have is so deep, is that, as you mentioned, it's a big enough problem to have even if this is your sole focus.

Yes.

And I think this is why we're putting this as our sole focus.

I'm not an analyst anymore, as you know, so I don't really give advice for free or for money. But I think that this answer convinces me, but I think it can be made more crisper to convince people with less awareness, even of how modern SIEM works, because you will see people show up at your door and say, yeah, I can achieve all that with my Splunk. I can achieve all of that with my SIEM du jour. Well, Splunk, probably not, because it's going to be a lot harder, but a modern SIEM, they would pick. So, probably, as far as free advice, probably make it even crisper. If you talk to CISOs, like, why can't I just use X? What would come up?

Yeah.

So one more direction that came to me when I was thinking about this episode is that we sort of had some attempts to do deeper application security analysis of sorts. And many years ago, we had this thing called RASP, which one of my unfortunate colleagues at Gartner have coined. The term went practically nowhere, and it was one of those stillborn terms that just didn't stick. It's like people who make fun of my former colleagues, Gartner analysts and Forrester analysts, for acronym coinage. This was like an acronym coinage that just flat hit the wall. Splat. No traction. But RASP, in some regards, is a little similar to this type of a deeper application analysis, deeper application telemetry collection. So, I mean, the reasons for its demise are long, but instrumentation complexities came up as one of the reasons, at least in my mind. So, why would ADR succeed? And why is ADR easier than RASP circa 2013, or whatever year it was? I don't recall.

Yeah. So I think that going back to your mention on RASP, I think that RASP didn't probably have the best traction due to, from what I've heard, 2 key issues. The first one was just how hard it was to implement. And second was, in many cases, the scope of the solution itself. It was very language-specific, service-specific.

Yes.

And it really, you know, put some people in a position which was, as I mentioned before, okay, I understand how a singular event in this service is problematic, but how does it, you know, combine to my broader application context? Should I care about this event? Is it going to lead to a potential impact on me? And other things like that. And I think that essentially, these are the 2 things that we believe ADR are a lot different, because first and foremost, as I mentioned, We believe that distributed systems is the hardest problem to solve. And this is why, you know, staying in the context of a specific service is not what we are trying to do. What we are trying to do is essentially to get the context on how a request looks like from the internet, through your CDN, through your load balancer, through your different services until it gets to the data, how it actually looks like. And I think this allows many teams to start thinking about, again, attack scenarios. And not just like a singular way to block an attack in the application layer or an edge on their WAF. And I think the second thing, of course, is about implementation. So we are not that dependent on language anymore. So we can come in and solve for multiple different languages. I think that the— in our case, we're actually utilizing the fact that many people already have instrumenting their application with application monitoring tools.

Hmm.

So we're just plugging into this in order to actually provide deeper visibility without asking you to do anything and help you identify your problems without even asking you to take this toll on your development teams. And I think that at least for us, you know, the ability to reduce dramatically the barriers for the security teams to enter such a solution, as well as the ability to proactively start defining attack scenarios rather than singular attacks, is what will make ADR stand out. And essentially, it will give back the control to the security teams to start understanding better how they can actually protect their applications.

I think that makes sense. And I think that an untrivial part of your answer seems to be that the technology has changed quite dramatically, and you can be language independent here. While whenever Joe created the RASP acronym, you really had to, hey, here's a Java library, here's a C library, here's the ungodly language nobody uses library. But I still want it because I'm a big company. So, this is better today, dramatically better. And I think this part is there. But does it really mean relying mostly on eBPF-type tracing, or are there more key telemetry inputs in your tech stack?

So, I think that, again, how we see ADR is it should be and is playing nicely with all of the different stacks you have. Our approach is actually ADR essentially needs to be connected to not only your security application security stack, but to your engineering and DevOps stack as well.

Yes.

Because they also have the amount of observability tools that companies already invested in, but unfortunately not utilizing is huge. And the ability to actually plug into all of this from the monitoring side to the just like operational side of things, to ingest all of this data and contextualize all of this data actually allows us to go way, way beyond the eBPF. Our solution in our case is a combination of, you know, OpenTelemetry and like tracing abilities with a bit more deeper ability to understand the OS. And we actually are able to not only correlate these 2, but take it into account with how your WAF is actually being configured, how your load balancer is configured, how your cloud is configured. But looking at this from, again, from an application flow.

So then the key inputs into your ADR stack is not just tracing and to an extent it's logs, but also You described other telemetry sources and also other context sources, because it's not just about the telemetry.

Correct.

Again, our goal is— I think it also strengthens the SIEM argument, counter-SIEM argument, is strengthened because some of the stuff, in most cases, SIEM would not know where to put, or they would put it in a place they can't use.

Yeah. And I think it goes back to— I can give an example. We just published research we had about an implementation-based vulnerability we called ALBeast, which really I think amplifies the complexity of modern applications and also the shared responsibility of modern times. So think about your authentication today. 10 years ago, one team, one room, probably built an authentication, maybe utilized some external things. Now you build an authentication in your service. So you have an IDP that is maybe doing the initial identity aspect for you.

Yes.

But then your CDN that might have some ZTNA features into it might also be part of your authentication flow. And then your load balancer in the entrance to your application might also change some of the things. Like in our case, ALB, you know, ALB, we're actually by feature offloading the authentication from some of the services themselves. And then you have several teams, each of them building another microservice with authentication. So, you know, if you have any problem with your authentication, who's to blame? Who's in charge?

I think that also has the— it makes the cross-team dynamic a lot more tricky. And I think that we've encountered this when we discussed cloud remediation. In the cloud remediation episode, we kind of had this discussion of why remediation is a challenge. It's not because your tool doesn't know the tech stack, but because 5 teams own components of that, and they may not all like each other, and there may not be a process you, the security side, is even aware of how they collaborate. So, I think that that part is not a good argument. I mean, it's probably much better against the SIEM than against CDR because they're also aware of the modern realities of modern IT. But I think that if you're talking about like, give me the logs, I'll give you the outputs, then it's a nonstarter. You would never— they would never get to the truth.

And here, Anton, if I go back again, going to some of the differentiation we are seeing with the CDR, and I'll ask you if it's okay. Yeah. Because going back to what you said, even in the cloud world, you know, the remediation is hard because you have so many different components of people. That are combining, I would say, one flow.

Yes.

In the world of application, again, and we call it like, there is a shared responsibility model essentially, right? So not only internally it's so hard because you have all of the application security and the amount of different engineering teams building those microservices or services, but also now you have all of these external to your cloud or external to your environment components, which are actually part of your application flows.

Hmm.

So if you have an issue, like, again, going back to, in ALBs, for example, one of the things that we found there was that it's not enough for your cloud team to actually change the configuration of your application load balancer. Your security engineer, like your engineering team, actually have to change their own code. So, you know, it goes through so many different components. If you have a WAF bypass due to the fact that there is some service drift, So who do you need to talk to? WAF is external to your application. It's like, it's becoming so complex today that we truly believe that, you know, you have to have like an ability to automate at least this view because it's too wide.

So let's elaborate on how your ADR tool stack correlates various data sources to get that type of beautiful view that SIEM cannot get. So can you share a few examples of badness that it would cover or uncover rather?

Yeah. So I think that first and foremost, the ground truth is about the code execution flow in the application itself. And this is why tracing for us is the bread and butter of our solution due to the fact that we need to trace both, I would say, horizontally to understand how a code execution flow works from the internet until the data, as well as within the application itself, from the initial function until it has essentially calls in the server level, container level, it doesn't really matter. And in order to do it, we actually managed to enrich all of the code execution flow that happens in the application level with those clusters and containers and cloud environment variables that are also composing the runtime environment of the application itself. And on top of it, what we are actually able to do is going back to the fact that we want to see the full application flow.

Hmm.

We are also integrating into your application security tools.

Mm-hmm.

The ASPMs you have, the SAST, the DAST, as well as the WAF and others, in order to show you kind of where other people found issues, what other controls you have to mitigate some of these issues right now, and essentially analyze this from an attack scenario standpoint to tell you, okay, but these are the issues that you are now susceptible to. And this is where we can help you mitigate them.

Hmm. It actually does sound like real work has to go into this. This isn't just like, give me the inputs, I'll shovel them in one database. There is a non-trivial amount of logic that has to be run so that this happens. That, to me, creates a unique value for the product, is that there is this logic in doing this as opposed to just brute force data analysis, pile it up in one place and search, for example.

Yes.

Yeah. So, this makes sense. And I think that of clarifies why this exists and I think why it should exist. So, you mentioned other tech, and I think AppSec is maybe uniquely confusing lay of the land specifically. So, like, there is this classics like DAST, SAST, and WAF. There's this fairly mysterious thing called SPM that nobody understands. So, how does this work with both the classics and the new elements? Like, for example, does the company need to have an ASPM before they buy your tool? I would guess no, but— Still curious to hear what you say. Great.

So I think that the— how we're thinking about the combination is that when I would say the regular AppSec is highly focused on process, on how can I help developers write better code and prevent this vulnerability from hitting production, we're more focused about understanding how the application works in the runtime environments and understand what attack scenarios they are susceptible to. So we are very much focused on the right side of this. Now, of course, in the combination of these 2, there are a lot of, I would say, benefits. We are actually integrating to these SAST and DAST and ASPM tools in order to understand what they identify before production and actually be able to help you understand how to prevent or if you need to prevent them from hitting production because of their impact. So we are actually able to analyze this and combine this with our attack scenario analysis.

Okay.

To tell you, look, this vulnerability, despite it being done medium or whatever, it's not really reachable due to the fact that it cannot be reached from the internet. It doesn't really touch sensitive data and so on. And from the other side, when we talk about the WAF, about the edge solution, we actually integrate to this in order to make sure that the WAF that you have is actually covering the application as it changes over time. So when there is a drift in your application—

Wait a second, you still need WAF? You don't claim to replace WAF.

So at the moment, ADR at the moment, I do think that, you know, WAF, there is a right to exist now, you know, but I think that where we are seeing a lot of problems sometimes with the WAF itself is that, you know, the ability or the amount of false positives that the WAF is creating or the misconfiguration and coverage issues that the WAF is having is very high. Because they don't have the context of how the application itself works.

Yes. And that's been a bane of WAF existence since its birth. Yes, very much so.

So with our integration to the WAF, we can actually automatically help you identify when you have coverage issues or configuration issues or help you triage better some of the alerts you have based on the actual context of the application itself. So we're actually playing very nicely with them.

I think that makes sense to me regarding WAF. I think that if you would eventually move, I think RASP also in that sense tried to say, we're going to block the attacks inside the app, and that made it a lot trickier and ultimately not succeeded. Yeah, that makes sense, the WAF. So given that we are starting to be short on time, I want to sort of ask you a broad question about the future of this space that you're pioneering. So what are your thoughts on the evolution of ADR? Where would it go? Like, what capabilities we can expect to see, and what sort of magic would materialize from, well, companies like yours?

I think the way we envision this is very tied to the, I think, the processes that are happening in the world of application security itself.

Mm-hmm.

As I mentioned, the challenge itself in the world of application is continuously growing. The complexity of the application, and I think the evolvement as well of attackers, is growing dramatically. And many people I'm talking to, you know, talking with me about the call to action, both in the ability to first and foremost just automate a lot of these processes.

Yes.

And 2, take a bit more proactive approach, rather than, you know, trying to prevent things from happening, actually be able to mitigate them from happening until you are able to essentially patch or whatever. And I think that what we are trying to achieve, and I think what ADR is trying to achieve, is exactly that, is the ability to allow your application security or product security team to do much more with their time by automating a lot of the context, by automating the ability to continuously do threat analysis and threat and attack scenario analysis on how your application actually looks like.

Mm-hmm.

To never be surprised by a new API or a new service that was introduced because somebody now will tell you this. And of course, the most important thing, to actually allow this team to understand where they need to focus their prevention efforts and their engineering team prevention efforts. While on the other side, have the ability to mitigate any attack that is coming until the patching cycle is done, which I think is kind of— this is where we envision ADR, like the ability to on the one side help you prevent the attack from happening, but on the other side, help you to mitigate it until you patch it and kind of correctly finish it if you can. Sometimes you can't, and we mitigate for you.

Yeah, I think guidance towards mitigation is critical because in many cases, especially for less enlightened companies, the remediation, the actual resolution is a lot of work by various teams who have to organize maybe in human time, meet in person, conference rooms and all that. So some form of temporary mitigation would probably be asked of you by clients immediately. Hey, you found this thing. You can detect that it's being abused. How do we stop it even for now? I think that makes sense.

Yeah.

So given the shortage of time, I want to ask our traditional closing question. So give us one resource to read. I added your blog on the ALB vulnerability to the resources already. And maybe one tip. I think in this case, one tip would be on the application security, how to make it better or something. Pretty much whatever you want, whatever you want, the resource and the tip.

So you mentioned, I think that the read on ALBs is actually interesting, not because it's ours, I think because it's a very interesting finding from our team on some of these complexities that we also, I think, learn as we go, how complex is this world of new applications. I think I would highly recommend it. I think there are several publications from Chris Hughes that I recently read about, you know, he was writing a lot about vulnerability management.

Mm-hmm.

And I think one of his interesting I would say, publications over the last few weeks were about the limitation of this and how to take a bit more proactive approach, which I think really, really shows both the challenges that you are seeing. As a person that is so much involved in the world of vulnerability management, I think he really, really managed to crisply tell where he sees the limitation of the approach. And on the other side, I think some of his ideas there, really talking about how to to take AppSec to the next level and take AppSec to a more proactive. So I'll be very, very, I think I'll highly recommend people to read, I think, some of his latest about it.

I'll probably link to his book on VM as well, right? That's probably one of the more interesting resources if you want to have a comprehensive look at this, right?

Yeah, this, and then maybe I can share later also with you the specific ones there, which are interesting.

Yeah. Perfect. Well, in this case, what about the tip? What about how to make AppSec better?

So I think the biggest tip I can give, I think, today, based on the recent finding we had, is I think that visibility is probably one of the most important parts in application. And I think that it's very important, or many, many times, I think application security is kind of, because of the shared responsibility, beyond the comprehension of some of the security teams or security professionals.

Yes.

And I think the tip that I'll really urge from the application security teams to their directors and CISOs is to really think about what attack scenarios can I really see and protect today in my application and what's not. Because I think this will unfortunately open a Pandora's box, but on the other side, this will really help them understand what they don't know.

Thank you very much for this. And I think that you're right about the shared responsibility is still screwing with people's brains, even though it's been, you know, 15 years old concept, but people still miss who does what and why and how. So thank you very much for this. This was very Sort of fun and passionate and admittedly a little long, but still good long, not bad long. So looking forward to seeing you in the industry.

Same here. Thank you so much, Anton.

And now we are at time. Thank you very much for listening and, of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikin and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.