EP110 Detection and Response in a High Velocity and High Complexity Environment

Security running.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts, as ever, are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts. as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. You can also find us at cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit the subscribe button in your podcasting app of choice. You can also rate our podcast if you like our content. You can follow the show and argue with us on Twitter as well, twitter.com/cloudsecpodcast. Anton, this is an awesome episode because we're hearing from head of DNR at a major and subject to many interesting attacks kind of organization. And it's not Google.

Yeah, yeah, because these guys have money. Yes. And people like the cybercriminals, attackers, tend to like money, many of them.

Why do you rob banks? That's where the money is.

Exactly. And so this company is certainly not a bank, but they are very close to money. And so people, I would assume threat actors, think, hey, they have money. I need money. I'm going to go attack them.

Who doesn't need money?

And you won't believe what happened next, but it isn't the episode title.

I love this episode because I had the pleasure of working with our guest back when they were at Google, and it's great to see how they've grown and matured and put their role into a bigger and bigger place. So it was really, for me, a personal pleasure to reconnect with this colleague. So perhaps with that, listeners, let's turn things over to today's guest. I'm delighted to introduce today's guest, former colleague of mine. Today, David Seidman is the Head of Detection and Response at a small stock trading app you might have heard of called Robinhood. David, first off, thank you for joining us today. It's a pleasure to see you again and have an episode together. Can you tell our listeners just about your journey to joining Robinhood and then how you chose what to prioritize your detection focus on once you got there?

Thanks, Tim. It's great to be here. So I'm the head of detection and response at Robinhood. Um, before Robinhood, I worked for Salesforce, Google, and Microsoft, and I actually worked with Tim at Google. So it's great to see you again, Tim. And Anton, great to meet you. I joined Robinhood about a year ago, November 2021, and started off managing the detection and response infrastructure software engineering team. And then over time, I picked up responsibility for the full detection and response team. And it was a really interesting transition because these previous companies I'd worked for were very focused on intellectual property protection and user data protection. And the attackers that were going after them were generally nation states. Whereas at Robinhood, we're in the financial sector. Most of the attackers who are coming after us are economic. They're looking to make money, to steal money. When they're going after users, they're not interested in things like GPS records or personal emails. They're interested in stealing money. So there are some nation states who are interested in stealing money, but overall, it's a really different attacker set. And that leads to a different set of threats that we need to be concerned about from a technical perspective.

Doesn't mean they're weaker, right? They can be as strong, right?

That's right. They can be as strong. Generally, the very highest-end attacks are nation states, but the next level down is very sophisticated still and also can be a little bit more customized. They're coming after us specifically rather than trying to Like, there— no economic actor, no economic criminal group has the goal of penetrating every organization on the planet, which, as I understand, some of the nation-state groups do.

That's a reasonable goal. Why not just get them all?

Aim high, moonshot, right? But for the economic actors, they're still really sophisticated, and they're coming after both our users and our enterprise at the same time. And so we have to think about threats somewhat more holistically in that regard.

Oh, now that's interesting. I hadn't thought about that. Are you also responsible for detecting attacks on your users?

No, my team is not responsible for that, but we do have a lot of overlap with it because a lot of attacks that we see could either come from user compromise or from enterprise compromise. And so a lot of things start with my team or start with the user-focused team and then transition between us and collaborate between us.

But there is a common manager, I would assume, right? So sorry for asking a possibly boring org question.

Oh, I think that's actually a really interesting org question.

Indeed, some of the attacks will cross over, and it would be a bit of a head-scratcher exercise to figure out, oh, is this a user attack or an enterprise attack? Kind of a little bit like with logs. Is it an enterprise log or a user device log? Sorry, Tim is smiling, and I know why. But how is it organized? Are they peer teams with the same boss?

We've actually gone back and forth on that, but they are currently part of the security and privacy organization. The interesting thing is that you can continue that spectrum from cybersecurity attacks to non-security attacks, things like bank fraud, users who do things like deposit money. And then we have a feature where when you deposit money, under some circumstances, you'll have immediate access to the money before the transaction clears the financial system.

Mm-hmm.

So one attack is users who deposit that money, get the instant cash, withdraw it immediately, and then the financial transaction fails. And so they've effectively stolen money from us.

Isn't ACH great?

Yeah. And there's so many different attacks like that that are purely attacks against the financial system and against the financial risk controls. So on the far end of the spectrum, those sort of attacks are not in the security and privacy organization, but we work across the entire spectrum collaboratively. One of the interesting things about being at Robinhood compared to my previous companies is it's 10 to 30x smaller than any company I've worked at before.

Oh, that's much smaller.

Yeah, it's tiny compared to a Microsoft or Google. And so you can just, you can know everyone, you know, maybe you, maybe you don't know everyone well, but all of the people working in your area, you can have at least somewhat of a relationship with. And anyone who you work with regularly, you can know the entire team really pretty well. So that's a really big change for me. It's pretty nice. Much less bureaucracy and politics.

Well, before we turn this episode into a recruiting pitch for Robinhood, we should ask the next question, which I know Anton wanted to ask because this is something he and I argue about all the time.

Yes, and in fact, it's kind of funny because we argue, we agree, and there's— I think there's a kind of a framework emerging out of this. So the essence of our discussion is the following. We do see people who want to just buy detections from a vendor, buy detection content, or buy a tool that has prebuilt content. So, like, more like a precooked canned model. But there's absolutely people who show up and say, what do you mean? You don't know my threats. I should be able to pretty much craft everything I want, fully bespoke and homemade. So, of course, we use the cooking analogy, kind of like a frozen food aisle versus a chef coming to your house, or you doing the cooking. So, does this framework make sense? And maybe when you look at your detection, how do you— Think about it. Do you care about detections made by the vendors? Do you cook everything at home? How is this working?

I completely agree with the framework. This was actually one of the interview questions I was asked when I interviewed at Robinhood. The way I think about this is my team of 5 or 10 is never going to be able to compete with a company that is doing detection and response as their main business. They have dozens to hundreds of engineers, and they have visibility across all of their customers. So there's no way that our small team can compete with a company that's doing this as their main business. For attacks that are seen across the industry, so for commodity attacks, for attack campaigns that span many businesses, there's no way that we can make internal detection that's as good as what we can buy in the marketplace. So for attacks that are not specific to our business, I think it makes very little sense for us to create our own But environment, what about the unique environment? So, for our unique environment, that's where our detection team comes in. The things that only we know about are the things that only we can detect attacks against, and that's what our detection team spends our time on. And even within that, there's some interesting prioritization questions. Do you focus on taking well-known industry techniques, like looking at the ATT&CK matrix, and try to apply those to your services? That's one approach. That approach tends to be kind of boiling the ocean. There's a lot to think about, and it's hard to work through it all. You can start with your red team and purple team operations, use offense to identify attacks that work against your infrastructure, fix and detect those things, do it again iteratively. Another approach is to use threat intelligence to try and figure out what's happening to peer institutions. Those attacks are likely to be applied maybe by the same actors against your institution. That said, that intelligence can be hard to get, and it can be hard to figure out how that applies to your own system. Folks also look at anomaly detection. We have not found anomaly detection to be terribly effective. I think that's true across the industry because there are so many ways—

Yeah, but let's go kick the ML unicorn while it's down. I think that's a fair point. Of course, Chad—

No, stop. He's already dead. Poor UEBA. It's on the ground twitching.

I think anomaly detection has its use cases, has its uses. I just don't think we initially, I think as an industry, thought that it would solve the problem. We would look for the anomalies, those would be the bad things, and we'd be done. I think now more sophisticated teams recognize that anomaly detection can help you confirm that something else that you think might be suspicious is indeed suspicious and worth investigating. But I don't think it works on its own.

No, I think that's a really good understanding. Could you explain to users what you mean by that last point, actually? Because I think that's worth double-clicking Right.

So when we create detection logic, we're never sure— very rarely are we sure that we've caught something malicious. Every way of detecting something comes with some amount of uncertainty, and for the most part, that uncertainty is pretty large. If you combine different indicators of suspicion, you can start to increase the amount of confidence that what you're looking at is actually malicious behavior. Anomaly detection can be really effective at augmenting those signals and adding another reason why we think that something is suspicious. And the reason why we want to filter down to the most suspicious events is that our operations team, the people looking at these alerts, only have a limited amount of time, and they get overwhelmed very easily. So, we need to be very focused in what we ask them to look at. Using anomaly detection, using well-crafted detection logic, can help them focus on the alerts that are most likely to lead to malicious behavior.

But sorry to try to kind of corner you a little bit, but of course the original question was kind of handcrafted detections versus vendor-cooked detections. And of course your sane and well-balanced answer is that you use both.

Of course.

But yeah, I mean, I'm not arguing with that at all. And I don't think me and Tim argue about that. It's kind of, we sometimes argue, I guess, about the—

The balance.

Maybe the balance or the priority logic. I think that your pillars that you presented make total sense. Commodity attacks, sure, outsourced to vendors. Typical environment elements outsourced to vendors, unique environment elements custom, unique business custom, but anything else that you consider when you decide, should I just copy an attack from a vendor, or should I just use the signature from a vendor X, whoever that is?

I actually think about organizational maturity a lot. One of the big advantages of a vendor is you can basically write a check, do a deployment, and get a lot of coverage pretty quickly. It's relatively simple. It's relatively off the shelf. By comparison, when you write your own detection, it takes a long time. Even if you have a really large team, there's still a certain number of pieces of detection logic that a person can write per month. And if you're starting as an immature organization, your team's probably not very large to start. So you need to hire people, ramp them up, and the time that it takes you to develop a detection base that you've created yourself is on the order of years, whereas the time that it takes you to buy things and deploy them is on the order of months. So I think that most organizations who are relatively early to moderate in the maturity curve are best served by starting with vendor products as they build out their detection team and start creating in-house collateral. And then that will naturally mature as the enterprise itself hopefully matures over time. The detection team will mature and keep up with it. And if you land all of this just right, you'll sort of run out of the low-hanging fruit in the vendor space right as your detection team is hitting full speed and full strength and is able to start creating lots of internal valuable detection.

That's the dream, isn't it?

That's the dream.

I want to shift your mind a bit away from dreamland, back to reality, and talk about, I hate to say it, AWS. I know you're running a lot on AWS, and there's a lot of vendors doing detection for AWS, including AWS themselves. So how have you thought about your vendorized detection approaches on AWS, if you have?

So maybe, sorry for jumping in with like, Cloud provider source for detections, third-party vendor, your team, right? 3 pillars. So balancing question again. Sorry, this is me, my former analyst brain.

Great question. So I think the first place you start is with the platform provider. They are, at least in theory, best positioned to identify threats against their platform. And in some cases, they have proprietary data that no one else has access to. I think it's also very nice to flip a switch. AWS has GuardDuty, Microsoft and Google have similar products. And you can kind of just flip a switch and start getting alerts. The problem is, there's a lot of noise. And so you have to invest a significant amount of time in tuning those alerts. GuardDuty has a reputation for being particularly noisy. It is pretty noisy, but everything is pretty noisy. So I don't want to make a direct comparison. I think the main point is it's free on the surface, but it's not free in practice. Taking it another layer deep, AWS gives you a lot of data in CloudTrail. And again, the other public cloud providers do the same. And there's a lot that you can do with that internally to create your own detection. There are a lot of vendors who consume that data and detect things on your behalf. So I don't really draw a distinction between the built-in detection in a public cloud provider and what I can buy on the open market. Ultimately, I have to write a check of a certain size. I get a certain volume and quality of alerts that cover certain types of attacks, and I have to decide whether the money and the overhead of tuning those alerts is worth the quality of the detection I get from them.

So there's really no free lunch. You flip a switch, and so it's operationally free to develop, but it's expensive to tune these things and get the noise out of the system and get value out That's exactly right. Yeah.

So, your real decision is if you have a million bucks, do I give them to kind of consultants? Do I give them to the cloud provider for some kind of enterprise security tool or a third-party vendor? You decide kind of, I guess, based on the situation, based on what you can get. Sorry, that sounds like every shopper and every purchase ever on Earth. So, maybe I'm not asking exactly that. So, there is no strategic perception that it must be X is better. It's really more about what you can get for the money.

That's exactly right.

Yeah.

Okay. Finally, I made it. So that makes sense. And I think, again, nothing super contentious, but it also— I love your rational approach here. So one tricky thing that we come up with sometimes is that people with expertise in cloud security, cloud threat detection is quite challenging. And of course, you are kind of a cloud native. So for you, maybe it's easier, but how do you approach the talent angle. If you say, hey, go write detections for my AWS environment, it has a lot of foundational skills they need to have. They need to know the cloud. They need to know security. They need to know cloud security. They need to know detection. And ideally, they need to understand threats to cloud. It's a kind of a tough ask, right?

And it's not just they need to know cloud. They need to know that cloud because each cloud is different.

That cloud, correct.

Okay. So here I may have a somewhat more controversial answer.

Yes.

I think that detection skill matters a lot more than cloud skill.

Ooh.

Yeah. So my experience is that you can teach cloud relatively easily. Yes, there's a lot to learn, but you can learn it as you need it more easily than you can learn detection. My experience is that if you take someone who has little cloud experience, it's pretty hard to find somebody these days who has no cloud experience. But if you take someone who has only pretty basic cloud experience, you can teach them the fundamentals really pretty quickly. There are very good training resources available. Typically, you will have teammates who can help them, and they can work on one area at a time and slowly build their knowledge while still being very productive in creating detection. When we have tried to bring people without detection skills into the detection space, so thinking here of somebody who has cloud skills but not much detection skill, They tend to oversimplify problems, make assumptions that risky behaviors will not happen except in malicious scenarios. And as a result, they write noisy detection signals that generate a lot of false positives and not as many true positives. So I think actually it's pretty clear, my experience has been pretty clear that you start with detection and bring people in and teach cod. I will say that it has gotten a lot easier over the past, say, 5 years to find people who have both skills. I've been hiring for my team recently. And finding good people who had both sets of skills was not that much more difficult than finding people who had the other skills we needed. You know what I mean?

To me, it's actually not controversial at all. To me, that's actually logical because detection is a much tougher skill set. It's a much tougher job than roughly understanding how to spell Kubernetes or something, or to understand how some of the cloud components work. I don't know.

I think I get tripped up by Kubernetes fairly often.

Well, there's that.

Just K8s.

Fair point.

Yes. So as you're ramping this team, what have you found to be effective in getting people up to speed? How do you actually, you know, like, let's accept this idea that it's hard to teach detection, but it's easier to teach cloud. How do you teach cloud?

So first, I want to say, I don't think that we have quite nailed this. I think we've learned some things, but I would never claim that we have solved the problem of ramping up detection and response people in cloud. That said, we've made some formal training available. And that has been very effective for the people who take it. We've really seen great improvements from classroom training, online learning. The problem we've had is that most people don't take it. Most people find that the day-to-day work that they feel like they're being evaluated on takes precedence over improving their own capabilities. And that's been true even when we've given people explicit OKRs to take training, we've put calendar blocks in place, we've checked in regularly. Still, People feel the pressure of the task that's assigned to them more than they feel the pressure of the potential for learning things. So what we found to be more effective is giving people hands-on experience, saying, here's a piece of detection we need that's related to cloud, go create it. And actually, most people don't even say, I don't know how to do that, because not knowing how to do things is just part of being an engineer.

Yeah.

And so, yeah, they might not know about that particular technology, But they'll go learn, they'll take whatever tutorial they need, they'll just fire up an instance and play around with it. And they're able to work through it. You know, the other thing is that the vast majority of use in AWS or a public cloud platform is really just a handful of services. So with AWS, for us, we're talking about Kubernetes, S3, RDS, which is basically just a database, Kafka, IAM, CloudTrail, There are a handful of concepts. For the most part, with the exception of Kubernetes, they're pretty analogous to technologies that we use outside of cloud. And so people are able to get started pretty quickly and build that baseline of knowledge pretty quickly. There's some esoteric stuff out there that's much more complicated, but you just really don't need to know that on a day-to-day basis. The other thing is that a lot of these core concepts are the same across different public cloud platforms. So even if someone's coming in having worked primarily on Azure, They will be able to work on AWS right away. We actually have hired somebody from Microsoft recently, and she's been able to start working in AWS with basically no transition time at all.

That makes sense. And I think that one thing that caught my ear, I guess not an eye in this case, was that you said that some of the data sources, some of the detection mechanisms are kind of the same. So I wanted to click on this, drill down into this, and maybe ask you about your favorite data sources for detecting things in the cloud. I would assume in Amazon, it would be CloudTrail and some others, but that's a big thing. So any more details on this?

I think you nailed it right there, CloudTrail, particularly IAM logs. We see so much attacker activity leveraging identity these days. That's really a great place to look. You can also see a lot of interesting access if you have your data stores configured in a way that segregates important data from less important data, you can get valuable signals out of that. On the other hand, if there's no easy way to tell whether an S3 bucket, for example, should be public, should have a lot of access or a little access. It can be harder to get a signal off of that.

But if you're an org that doesn't have a mechanism for tracking which buckets ought to have sensitive versus not sensitive data, you've probably got more fundamental problems than DNR when it comes to your cloud security.

That's accurate. Although to some extent, DNR is all about the security problems that the rest of the org hasn't solved yet.

This is such a depressing And inspiring view. This is like, I think you created some line that's simultaneously depressing and inspiring. Yeah, I don't know how can it be.

I really like that. DNR is about all the security problems that the rest of the org hasn't solved. That's great. That might be the quote of the quarter for the podcast, actually. That's—

I think so too.

So it's unfortunate that on the cusp of such brilliance, we have to wrap up. I'm going to ask you the traditional closing questions. One, do you have a tip to help people improve their DNR in cloud? And 2, do you have recommended reading so we don't leave our listeners empty-handed or empty-eared? I think as Anton might have said just now.

Sure. My tip, I think exercises are really undervalued.

Like sit-ups, push-ups?

Actually, that too. I'm a runner. Running is my passion outside of work. But no, I'm thinking red team exercises and tabletop exercises. Red team is a fantastic simulation of real attacker behavior. but they're expensive and hard to run frequently. Tabletop exercises you can run pretty often and you can test out a lot of different scenarios. And really, it's the only way to know how effective you are in practice. They're the only real attacker you can get, and it's the only way that your IR team can practice for, quote unquote, the big one and be prepared for those really catastrophic scenarios that you hope never happen. The other thing that's really advantageous about exercises is they're great for bringing awareness to executives and other stakeholders who are not maybe familiar with the state of security. My experience is that as you run the first few exercises, the executives respond, okay, this is a problem, go solve it. The second one, they say, why haven't you solved this yet? The third one, they say, okay, seriously, why have you all not solved this? And then about the fourth one, they say, oh, this is really hard, isn't it? Here's some more money, go solve this problem in the next 5 years. And then we come back 5 years later and say, yeah, we're better, but So, exercises. If you're not doing them, do them. If you're doing them, do more of them. And then your second question was recommended reading.

Yes, please.

I don't have anything specific to security, but I think a lot about teamwork and how teams can be effective. And so, I have some books to recommend along those lines.

Please.

I really love the book Radical Candor. The central theme of that book is basically tell people what you think. Politely but openly.

Politely?

They do add that? They do. Oh, absolutely.

Anton, just for people like you.

I'm sorry.

Yeah, yeah, I recommend reading the book because it's easy to take the wrong thing away from the headline. I like Daring Greatly. It's a Brené Brown book. She has the same theme in a few different books, basically again about being open. Her work is more about being vulnerable and being willing to hear things that make you uncomfortable, which I think is the flip side of radical candor. And radical candor actually covers the same concept. I like the book Extreme Ownership.

Mm-hmm.

The, the principle of that book is basically, don't think of any problem as someone else's problem. Think of every problem as your problem. And it may be that you need to go work with someone else to get help solving that problem, but don't think of it as we're dependent on them and we're a victim. Think of it as we have an active role in solving this problem. And then the last book is called Drive by Daniel Pink. It digs into what motivates people. It turns out that when you have above a certain amount of income, sort of a reliably sustainable income, that people stop being motivated by money and start being motivated by purpose, autonomy, and mastery. And it really changes the way that I think, and I think that other people should think, about how to manage folks and how to work with their teammates to help them achieve their best work and be motivated and love their jobs. I got into management because I want to help everyone love their job, and I think understanding what motivates people is really crucial in doing that. So those are the books I recommend: Radical Candor, Daring Greatly, Extreme Ownership, and Drive.

Perfect. Well, David, thank you for such a wonderfully structured answer to that. Most people end up rattling off whatever's off the top of their head. And it's clear you put a lot of thought into that one. And I, I for one really appreciate that. Thank you for joining us on the podcast today. I really enjoyed our conversation. And listeners, I hope you did too.

My pleasure. Thank you for having me.

And now we are at time. Thank you very much for listening and of course for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter, @Anton_Chuvakin and @_TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Bye.

Security. Security.