EP281: Deceiving Adversaries at Scale with Kevin Conley

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your host here as ever, myself, Tim Peacock, a PM on Google SecOps responsible for a lot of our agentic capabilities, as well as Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as on YouTube. youtube.com/@cloudsightpodcast. If you enjoy our content and want it delivered to you piping hot every Monday, please hit the subscribe button on your podcasting app of choice. Anton, we are talking about truth-telling? No.

No, the opposite.

We're talking about deception. The opposite.

Yeah. And I'm going to make the agonizing face a few times during this episode, not because I don't like it, but because it's been something that just hits a particularly odd spot in my soul, in my psyche.

Yes.

Namely that it's something that everybody agrees and nobody does. I mean, you can say that patching fast is also something that everybody agrees and nobody does. And I guess I'm just—

How about good asset management?

Or good asset management? No, not good. Correct asset management. I mean, good wouldn't quite cut it.

Asset management in general, perhaps.

Yeah. Or even—

yeah. No, this one hit a spot for me too, because years ago I wrote a really interesting PRD in this space that nobody was willing to be like, oh yes, that would actually materially improve security. Couldn't get it funded. Broke my heart. And so the other funny thing for me with this episode, Anton, is when I was early at Shape, people thought Shape was a deception vendor.

Yes.

And we were not.

Yeah, because you used some of this in your messaging at the time, which was like the anti-bot argument about confusing people, bots, whatever. But it was a very interesting side journey into deception.

Yes.

And so maybe listeners, without any further ado or resistance, let's turn things over to today's guest. Today we are joined by Kevin Connolly, the head of deception tech team at Riot Games. Kevin, thank you so much for joining us today. I love deception as a space. I think it is endlessly fascinating. One of my favorite board games of all time is Secret Hitler, which is of course a bluffing and deception game. So this is great. Never thought I'd get to talk about Secret Hitler on the podcast. Listeners, if you're listening, my friends, I'm not Hitler. Security practices have seen deception tech before. This isn't like a totally new idea, but it feels like things are different now. And so could you walk us through what's different about cloud tech that makes this kind of the compelling moment for this practice and why other people should maybe think about getting some momentum behind deception tech these days?

Yeah, absolutely. I think a lot of it comes down to the approach that we're starting to see deception vendors and then deception practitioners take as it comes to you know, what they're putting out there and kind of the goals of the programs. So for a long time in the beginning days of deception, going way back, you talk honeypots and putting intentionally vulnerable resources out on the internet in order to generate threat intelligence data. Still done today, still can be useful in certain scenarios, but I think more and more we're starting to see practitioners look for faster detections and maybe coverage in areas that you wouldn't otherwise be able to achieve coverage. So it was funny, actually was listening to one of your episodes this morning, a new episode this morning, and one of the questions that was posed was, what devices do you have that you can't put an agent on? EDR, MDR, whatever your tool may be.

Yep.

And I think that's actually one of the really strong use cases for deception. Deception in its simplest form can be a very small amount of text, and if you can put that somewhere, you can cover an entire environment with deception. And I think that is becoming increasingly powerful in a world where it's difficult to know all of the areas that you can deploy your tools and then deal with the data that they send back?

So I'm in a really weird position because I was involved in many early attempts. And I guess Tim would make fun of me for being, quote unquote, the proverbial old guy. But yes, I do recall the early Honeynet Project efforts going back to— well, Lance Spitzner started it in the late '90s. No, I wasn't there in the late '90s. I'm not that old. But early 2000s, yes, unfortunately. And there were sort of a few waves of vendors. Going back to the man trap in the, well, early 2000s and then another wave. And it's just that what I've noticed over the years is that the people always have a very positive view of that and just never do anything about it.

Mm-hmm.

And I'm thinking about the history going back, maybe not quite 30 years, but if you go look at some research papers, they're probably research papers from the early '90s about how deception is a good idea. And some people kind of reinvented honey tokens a few times, which is the bits of data that you leave to attackers. So, this has this weird thing that if you ask people, they say, oh, wow, great idea. But if you look for deployments, you have exactly nothing. And vendors die like flies. Well, the latest batch have died, and then now they think there's a latest, latest batch, probably like a gen 3 at this point. I'm just really torn about it. Like, I mean, emotionally, that it's a good idea, and we perennially don't do it. And so, is anything changing? Well, you are doing it, clearly. So, there's, you know, thanks to Canary, is, of course, very much alive and well, and people are doing it. And this is being— having a— A very successful track record, but mystery.

Yeah. What gives, right?

Listeners, Kevin, are you two wondering if Anton actually asked a question there as much as I'm wondering if Anton asked a question there? I'm not sure what the question was.

Okay. The question was, is this really happening?

Yeah.

Is deception really happening this time? Because there were a lot of false starts. Okay. That's a short one.

It was a deceptively long question.

Let's say that. Well, and I think the reason is there's no easy answer. Is deception around to stay? I mean, I certainly hope so. A little biased maybe, but I certainly like to think it's around to stay. I think one of the arguments that I make for deception, and one of the reasons that I really do hope that it's around to stay, is a lot of times defenders find themselves playing a catch-up game. Attackers find some new tool, technique, whatever it happens to be, and defenders have to react to it. And deception, in some ways, can turn that on its head. If you look at the MTrends report from last year, I think it was 17 or so percent of initial vectors was via some form of phishing, whether that be emails or voice. And if you look at cloud, I actually think that number was higher, though I don't remember the exact number. Why is fooling somebody in the wheelhouse of attackers and only attackers? How come defenders can't fool someone as well? I actually think that that's one of the things that drew me to deception after having practiced other areas of security for some time is the creativity that it gives you, and it allows you to protect your environment with what you know about your environment. If you give up home field advantage, you're leaving a lot on the table.

I think that's a really key point, is people always talk about the inherent asymmetries. People love to say, oh, defenders have to win every time, attackers only have to win once. But first, that's not true. And second, We forget that we do have massive home field advantage because we know what our environments look like, or we should know what our environments look like. We don't always.

Yeah. Better defenders do versus defenders don't, right? Like there's this—

That's true.

Classic examples, right? Yes.

Yeah. And, and even if you don't know it all, this is something that you do know, right? When you're putting it there, at least for some amount of time, this is something that you know better than anybody else. And actually that can be one of the trickier sides of figuring out how to implement deception within an organization is how much do you share to the rest of your employees? Is this something that you keep on the security team only, or is it something you try to message out as wide as you can? I don't necessarily think that there is a correct answer there. I know that I've had my own approach in the past, but mileage may vary and others may disagree.

Okay, so this takes us down a good path of like practical advice for an organization that's thinking about going down this path. What are the first things they need to do other than like know themselves? What are the first 5 things they should think about when they're getting ready to go down this path?

Yeah, the first one I think is an important one, and it is one that if, if you don't think about this, you will be disappointed in deception. And that is set expectations around what deception can and cannot do. To all the vendors out there who I've worked with and all the folks who I've chatted with about the cool products that they're building in this space, I apologize in advance. None of you have 100% signal, 0% noise. It doesn't exist. There is no deception product on the market that does. And I work with some good ones, I really do. But if you go into building a deception program with the expectation that you will never get what folks might consider a false positive, and maybe that's another tangent that we could go down later, I just don't think it exists. And having practiced it for a little over 3 years now, I've yet to see it. So that's number one, is set the expectation for the team that's deploying and the team who might be triaging alerts that come in from it, maybe they're the same team, that that's not what you're aiming for because I just don't think it's possible. But then, you know, Tim, to your point, the other part is knowing your environment, knowing what you're trying to catch. I think there's a difference in trying to generate threat intel versus trying to catch attackers quicker versus using deception to enrich existing alerts. I think all of them are valid approaches. I certainly would not advocate for Let's throw away the EDR and replace it all with deception, right? This is part of a balanced breakfast, if you will, of security. It's something that has to work in tandem with other tooling.

Yeah. I think that coverage was always the problem. And again, I recall for having covered this at Gartner in the past, I do remember the questions about coverage coming up. And people say, but with deception, I can never guarantee coverage. And I'm like, yes, that's right. We are not suggesting you toss EDR, NDR, and whatever else. Like, It's not about coverage. And if you complain about coverage, you're really complaining in the wrong place. Sorry, this gave me a flashback of trying to write a paper when an analyst and people say, Anton, why are you writing about this stuff? Nobody's deploying it. And I'm like, maybe they should. Maybe that's the problem. They're not deploying it. And of course, the enlightened organizations did deploy it and do deploy it. Okay, fine. Stop ranting, Anton. We want to talk about operationalizing more, like starting a program and doing other things.

Yeah, so I think when we come to the question of, hey, you know, let's work on the assumption that there's no such thing as 100% signal and 0% noise, there should also be an expectation that alerts will go off even in the absence of an attacker. I think if you deploy deception and in any environment, take your pick, and within I don't know, 3 to 6 months, you have not seen an alert come through that was an employee who got confused, you might have picked the wrong place. And the reason that I say that is, generally speaking, I assume that attackers are going to do the things that the employees probably were doing anyway. If they compromise an engineer, they might be looking at code. If they compromise somebody who works on your finance team, they might be looking at at financial information. If the real employee doing that real job could not find that resource accidentally, I don't think an attacker is likely to. And so using your own employees as a barometer of where you have actually put your deception and whether or not it's in the right place, I think is also a good thing to keep in mind. Now, that's a tricky balance, right? You don't want it to be overly burdensome on the security team that deals with the alerts, and you also don't, as much as possible, want to disturb the people who are actually just trying to accomplish their jobs. And so how you manage that within your own environment, I think there's, there's a couple of different strategies, ways that you can put things in the right place but look wrong to somebody who actually works there and maybe has a little bit of inside knowledge of what they should and should not be doing. But that's another thing is, as much as possible, putting it in the right place or the place that you expect the, the employees who actually do that job to work.

Which reminds me also that the discussion that sometimes comes up about what you described applies to both classic environments with, like, servers in racks, a lot of, like, legacy systems, but also to modern environments like cloud, where things are deployed, used through pipelines. Would you say that deception is more amenable or aligned with the modern environment, more with classics, or it doesn't matter? I think that I'm gonna— if I were to put words in your mouth, I would assume you say it doesn't matter, but like, probably I shouldn't do it.

I think that's a great question.

Yeah, I, I'm not sure if it does matter, but I think that we've had as much success— I mean, if you look at raw numbers, we've had more success deploying in cloud environments than what you might consider traditional environments. And I think the reason for that is anything that you can do for real can be turned into a decoy. There are, of course, vendors in the space that help manage or make it easier, but if you think about it, what is the difference between writing some Terraform that spins up a couple of resources in your environment that actually do a job and Terraform that spins up resources that are not intended to do anything, and therefore you monitor their logs more quickly or more closely? And so that actually kind of goes back to my earlier statement on the creativity is you are not limited to what deception providers give you. If you want to treat a certain string within your environment as a decoy, a real credential for something, a program in your SSO provider, any of it, it can all be deception. It really just matters how you monitor it and what you expect that use case to be.

That's really interesting to me. And so I guess, you know, we've talked about a lot of different techniques. I'm wondering which you think are the best to start with. You know, is it, should we spin up some fake resources, some fake files? Where should somebody start? Like what's the day one thing to do?

Yeah. Day one. Well, I can tell you what I did and why I wouldn't do it again.

Oh, good.

Oh, wait, wait, wait. What?

No, this is like, we had a whole episode about learning to fly planes by studying plane crashes. So tell us how you crashed the plane, man. This is great.

Yeah. So when I took over the Deception team, which didn't really exist before, so it wasn't really so much of a takeover as it was a founding, I decided to go for environments that did not have much, if any, coverage from security tooling on the premise that, okay, I'll increase coverage to an area that we don't currently have, and that will be a good thing for us.

Mm-hmm.

And although I still think that that intuition was correct, it also happened to be an area that the noise ratio was going to be high, and I knew that from the start. And my hope was that In doing so, I would be able to show that, hey, these tools work, right? You know, we're getting alerts from them. And simultaneously, here are some of the things that you can do when an alert comes in to make sure that, you know, we're not actually popped, because that's at the end of the day what our goal was, right? So how can you take one of these? So person finds the decoy, interacts with it in some way, alert comes through. How much work can we do between that happening and a human who might be sitting on the other end such that they very rarely actually see those alerts? And it turns out that we didn't do a good job of that middle part. And so a lot of alerts ended up reaching people, and it contributed, I think, more to alert fatigue than, than we probably would have liked. So what I would do differently is pick an area That is simple to deploy and that you feel you can get as close to 100% coverage in that environment as possible. So if you want to deploy API tokens to a workstation, how good is your fleet management system? I can tell you, I don't think I've run into a good one yet. I don't think I have 100% coverage, but I don't necessarily think that that's a fault of the deception tooling. I think that that's actually just highlighting a difficulty in reaching coverage across a fleet.

So that gets back to the importance of fundamentals. I mean, if people can't manage their fleet and don't know where their stuff is and don't know what good looks like or don't know what their assets look like, this is going to be another thing which is hard to do.

Absolutely. And in some ways that makes me feel better because if it were so easy to do this and nobody was doing it, it would make me wonder what we've been doing in the meantime. But at the same time, of course, it's frustrating because it means that I expect to have similar, if not the exact same blind spots as my colleagues do in the things that they work on.

So I want to have a quick side comment to this before we go to the program maturity discussion. There is a certain— what I heard in your past response is that there is a foundational level of being good at quote unquote the basics that people hate, fundamentals, whatever, before which it would be kind of stupid to do deception because you should really get back to like asset management and, you know, patching and whatnot. Of course, patching is in the news right now because of the Mythos and all this fun other stuff. But how would you judge, are your fundamentals good enough that you are ready to progress to deception? I'm sorry if I'm going all philosophical on you, but it is important for a lot of people who tend to sometimes jump too early. And maybe that's why some of the vendors failed. They got a lot of low-maturity clients that ultimately never saw value.

This is a question that I certainly think about a lot, though I'm not sure if I have the best answer. The way that it makes sense to me is I try to think of some of the security measures that an organization might take as the food pyramid that we were all taught when we were kids.

Yeah.

And on that bottom level are some things like have passwords, right? If you don't have passwords, like have an environment that you're protecting and not this kind of amorphous blob. have an external and internal environment that you can define. And whether or not you know exactly what the internal and the external are, maybe not as important. But I do think that one of the barriers to deception that folks run into is they feel as though they have to be at a certain maturity level in order to make use of it that, that probably doesn't exist.

Hmm.

I actually think if you have an EDR, or if you have any security appliances, you're probably mature enough for deception. Now, whether you want to invest in deception over one of those other appliances, that's probably a question that has the devil in the details. But I don't think you have to have the best EDR in the world. I don't think you have to have 100% network coverage and all these crazy fancy tools in order for deception to be a worthwhile endeavor for you.

I like that answer a lot because I also would've shared the perspective of, yeah, sure, you can do deception when you've got everything else covered. But I agree that there's value in it before you've got everything else covered. And I would probably annoy Anton by the list of things I think you could do deception before doing, but I'll leave that for a side conversation.

No, actually, I think it's the opposite. I just want to agree with that despite your answer being kind of unscientific in terms of how specifically what should be covered. But I think that if you If 12 reliable witnesses look at your environment and say, yeah, you're okay. You're not like Google grade. You're not top-tier tech company grade. You're not top-tier bank grade, even though that varies, but you're kind of okay. Intuitively, you're not bad. You're not great. Then you can do deception. I think that's a non-scientific but ultimately honest, correct answer is that it's lower than what people think. And I think what I heard from the Think guys A lot of their stuff is deployed in very mainstream environments, not by the elites. Okay. I don't want to— sorry, this is not a vendor show, and I think that they can pay us in beer later if they choose to do it.

So, I do want to ask a question to the team lead aspect of your job, which is, surely you're reporting metrics and understanding the maturity of your program. So, what are the key metrics you report to your leadership, and how do you think about the maturity level, and how should somebody else, you know, Imagine modeling their deception program maturity.

Yeah, well, I think earlier you mentioned learning to fly a plane from studying plane crashes. I think simultaneously we're learning to fly the plane while we're in the air. So to some degree, the metrics that we report up are being discovered and/or iterated on as we go. But some of the metrics that we find to be valuable to us are what percentage of an environment, at least that we're aware of, do we feel we've covered? So as you could probably imagine, not every decoy resource lasts forever. API keys can have time to live. And if you know that your fleet is 10,000 machines, but you only have valid API tokens on 7,000 of them, you've hit 70%. Is that good or bad? Is that number going up or down? Different questions. But knowing what that number is and tracking it over time, I think is worthwhile. Another one is kind of going back to the alerting scenario. What is our signal-to-noise ratio? I don't necessarily think that not having caught an attacker is a bad thing because you don't know whether or not you've been popped. If you've been popped 50 times and you haven't caught any of those, then maybe you need to reevaluate where you're deploying your deception. But the absence of a true positive alert isn't necessarily a bad thing. At the same time, And we draw this distinction internally. It's something that our team has kind of standardized on. The difference between a false and a benign positive is really important for us. So I want to have benign positives, and the way we define those is, after a person did an investigation into an alert, did they feel it was worth their time?

Hmm.

Was it a weird thing that a user did? that was non-standard and would have otherwise triggered user behavior analytics if we had a robust enough set of data on them. If those things are true and the analyst can look at it and say, yeah, it wasn't necessarily malicious, but like, that was weird, that's what we consider a benign positive. And those I love to see. If it was a false positive, like truly could have been, you know, was a waste of the analyst's time or could have been closed by automation, or maybe our tool just malfunctioned, Those are the numbers that I want to keep lower. So taking a look at not necessarily true to false positive, because I don't think that captures quite enough information, but benign to false or true to benign, those things are metrics that we find valuable and we've been using to the extent that we can. But then the last one is, if you're fortunate enough to have a red team or an offensive in-house, run them on your deception, see what happens, right?

Yeah.

And so Whether that be an in-house team or you bring in an external party. And I think that there's a lot of different ways that you can test this. In fact, there was a research paper written a bunch of years ago, the Tolerosa study, if anyone's interested in, in looking at it. But it basically set up a CTF with 4 different groups. Some groups were told that there were deception, other groups were not told that there were deception. Within those subgroups, some actually had deception, others did not.

Oh.

And then the study was to see what effect did that have on, on attackers and How quickly they were detected, whether or not they were successful. I'm not going to advocate for running your own study of that kind, but those are the types of things that I think you can really make use of.

That's a fascinating study. I really— I'm going to have to go find that one. That's great. If I find it, we'll put it in the show notes, listeners.

Yep. Same here. I was kind of fascinated by this because there was not enough data on this being studied robustly pretty much anywhere. Okay. So this is like— I'm going to channel my inner Tim and say, I hate to do this, but it's time to close the episode. And we usually end with 2 questions we ask everybody. We spring them on people as a surprise, but if you've heard other episodes, of course, they're not a surprise. One tip and one resource. And Tim would always say that the resource cannot be my blog, and I have probably not written about deception for at least 10 years, so I'm very safe from that. But any one tip and one resource, please. Or one— did I say resource? Tim, I shouldn't have said resource because that would steer Kevin away from poetry and stuff.

I like to say reading, and the reading could be anything from Mary Shelley to Mary Oliver.

Yes.

I do like reading, and in fact, I actually have here the 2 books that I would suggest. And so maybe, maybe these are just as much a tip and a resource. So, The Cuckoo's Egg by Cliff Stoll. This is a book that I think really embodies what the attitude and approach that a security professional should be taking towards their job. It's a fascinating story. It's a true story as well, which I think— so it's not a new book. It's certainly not the new hotness, but I think It really gets to what it means to be a successful security professional. And then Freakonomics by Steve Levitt and Steve Dubner.

Of course.

Again, not new, but I think it, for me, it was an inspiration for asking the right questions and not being satisfied with answers that don't explain the original question. And again, I just think it kind of speaks to what it takes to be in the security industry and be successful in it. So yeah. Those are my, my 2, uh, my reading points.

Fun fact, years ago we had my college professor, John Savage, on the show. I was his head TA for a cybersecurity class, and 4 years later, Cliff Stoll's daughter was the head TA for that same class.

Wow, that is cool.

So it is a small cyber world out there.

Oh, that is kind of freaky.

Yes, it is freaky. Uh, Kevin, thank you so much for joining us today. I enjoyed this talk, and still, I am not Hitler.

Awesome. Yeah, I appreciate it, guys. I really enjoyed this.

And now we are at time. Thank you very much for listening and of course for subscribing. Please subscribe so you can get new episodes piping hot. And if you love our content, please drop us a review on your platform of choice.

You can find this podcast on YouTube, Apple Podcasts, Spotify, or wherever you get your podcasts. Also, you can find us on our website, cloud.withgoogle.com/cloudsecurity/podcasts. You can argue with us on the Google Cloud Security Community site, googlecloudcommunity.com.

You can follow us on X, x.com/cloudsecpodcast. Tweet at us, email us, argue with us. And if we like or hate what we hear, we can invite you to the next episode.

See you on the next episode of the Cloud Security Podcast by Google.

Security. Security.