Tales from the Trenches: Using AI for Gmail Security

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks so much for joining us today. Your hosts here are myself, Tim Peacock, the product manager for threat detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and member of the cloud security team here at Google. You can find this podcast wherever podcasts are sold, as well as at our website, cloud.withgoogle.com. If you like our content, please do hit that subscribe button. You can follow the show and argue with us on Twitter as well, twitter.com/cloudsecpodcast. Today's episode is a fun one because we are talking about AI, we're talking about security, and we're talking about some of Google's largest properties. So this is a really cool episode with a peek behind the curtains of how Google does or uses AI for some security outcomes. But Anton, you're usually kind of a skeptic about AI and security. What's going on there?

So I would say that over the years of studying this domain, my attitude or my stance kind of evolved to being a short-term skeptic and long-term enthusiast. To me, I am definitely seeing the future behind it, but I'm not always seeing the today state. So funny enough, I was just looking at social media, and I've noticed something quoted Believe it or not, on Bruce Schneier's blog. And in this report, he had a really fun quote about ML, machine learning. And the quote was, we anticipate that machine learning would provide incremental advances to cyber defenders, but it's unlikely to fundamentally transform the industry, barring additional breakthroughs.

Oh, that's interesting.

So that's really resonated with me because I feel like today I kind of joked that ML unicorn cavalry isn't coming. Like, you're going to get improvements. But you're not gonna get magical things happening to your cyber defense posture in the short term.

So we're not gonna wake up at dawn with the AI Rohirrim coming over the hill. This is not coming.

Correct, correct. It may happen in 10, 20, 30 years maybe, but it's hard to guess. But to me, the short term is we're gonna get utility, we're gonna get value from ML/AI, but you probably won't get magic.

Yeah, well, that sounds about right to me. Curiously enough, I think in this episode we actually do talk about Both sides of that coin, ways in which it can be incremental as well as ways that it can solve problems other things are not suited to solving.

That makes sense. So really looking forward to it. And I know I would miss the recording, as you'll see in a few minutes, but it's a very fun episode and I loved it.

It's only fair that I hosted one by myself since you hosted the CISOs by yourself when I was having network trouble. So with that, let's welcome today's guest. Andy, it is so good to have you joining me today. I've been looking forward to recording this for ages. It was originally going to be a pre-cap and now it He treads firmly into the recap stance. So I guess we get a lot of nice retrospective things today. This is going to be a fun episode because we get to talk about the consumer side of Google a little bit and cross the streams, right?

That's right. That's right. I sit in a group called Counter-Abuse Technology, and we power all of the protections for user safety across Google. And we define user broadly, both consumers and enterprises. And we do a broad outlook on what are the threats to both of these kinds of customers, which quite often are quite similar.

Yeah, so we're talking like billions of people we're protecting here. That's pretty exciting. So your talk at the security sessions was mainly around AI, and AI is always a hot topic unless we're headed for AI winter. So tell the listeners, what are you doing with AI and how does that help with the mission of protecting billions of users?

So first of all, I often like to say use AI and machine learning only when you have to, and the reason for that is It's complicated to get it right. It takes a lot of work. So if you can solve your problem more easily a different way, please do that first. And to protect Gmail from spam, we do lots of what I just call counting stuff, right?

Uh-huh.

Reputation counting. We do tons of that. It's very effective. But when we really want to go the extra mile and protect users, not 90% protected, not 95%, 99%, 99.99% protected, that's where we really lean into AI. But you need to target AI to very specific problems.

Like what? What kinds of problems is it good at?

We deconstruct the phishing malware impersonation problem into probably a dozen different subproblems. I'll give you an example of a subproblem. We could have an AI model to determine, is this page phishing? Is it trying to pretend to be a reputable brand on the internet? But you can do that with a visual model, right? And so you can run like an AI visual model like, Hey, is this pretending to be one of the top 20 brands on the internet? But it's not. But you might have a second phishing model which actually collects the intent around the phishing link, because oftentimes someone's trying to get you to click on the link and say, quick, check out this great video, and it has a link. And we're like, oh, we'll do some statistical analysis and NLP and discover which are the words that really are correlated with phishing links.

Huh.

So even in phishing, We deconstruct the phishing problem down into multiple subproblems.

Ah, interesting. So you're able to break down this problem and then use the appropriate tools at each stage of the chain?

That's correct. And this is classic engineering. How do you solve complex problems as an engineer? You deconstruct it as a set of multiple problems and then you piece it back together again.

Well, that makes a ton of sense. So broadly speaking, what kind of security problems are good for AI to get a handle on and which ones are harder to address with ML?

Well, the biggest challenge you have with any kind of machine learning is, and I love this when I talk to any of my engineers or data scientists, well, do you have a label set of good versus bad? To the extent that you can collect thousands or tens of thousands of examples, that helps.

Is that the threshold? Thousands, tens of thousands? How many do we really need?

I think the more complex the model is, the more samples you need. So an example is if you're training just a basic logistic regression model, you might be able to get away with hundreds. And low thousands. But if you're training an image model, you probably need tens to hundreds of thousands, and those are hard to get.

Mm-hmm.

Because phishing emails, while there are plenty of them out there, you need to get enough samples, and getting each sample can take a lot of time. And we can kind of get into some strategies of how to do that. The second is that the definition of what's good versus bad is fairly clear. Even if you send it through a trained reviewer, that they're able to go, that's a phishing email and that's not. Or that's a mean message and that's not. And the less context-dependent that it is, the better.

Interesting.

So not only do you need enough label data, the definitions need to be clear and not necessarily contextual.

Got it. So it would be hard to train a model on, say, PM-to-PM comments in docs because that's super unclear whether something is mean or not.

Absolutely. And in fact, sometimes you kind of have to know who the person is, like, oh, that's a mean comment because I know his style or her style.

Yeah.

Is that sarcasm? What was this in response to? And so that's where it gets challenging to tell the nuance in, say, hate speech as an example.

That sounds very tricky. So you've been at Google a long time now, haven't you?

You know, it's gone by so quickly. 5 years.

5 years. In those 5 years, is there a time where you've been really surprised by ML being good at a particular problem?

You know, usually I'm surprised how bad it is. No, actually what happens is you go through this cycle with ML. You first start off very excited that like, oh wow, we could try using ML for this, and you get excited. And then you have what I call the trough of disillusionment where you're like, this doesn't quite work and it's really quite challenging. I'll use visual AI as one of these areas where like we struggled quite a bit to essentially put eyes on our Gmail classifier. And so we got very excited to do this. And we went through several quarters of trough of disillusionment where it was very challenging. But we were able to overcome a lot of the hurdles. And I go back to this challenge of our innovations often don't have to do with innovating around the algorithm. We actually innovated around the data collection—

Huh.

—and creating the training sets. It was really difficult to collect enough visual samples But we figured out a way to generate samples in slightly different ways so that we can augment our data and make it work for us.

Oh, that's interesting. So you stretched a sample of data by doing something tricky to it?

Yeah, we basically used AI to generate a training set for us in a way that was somewhat realistic. We used AI to sort of understand the variations that there were in our current dataset and to fill in variations that would be meaningful. So this is AI using AI to make more AI in a way.

That's pretty cool. So that's taken me a second to digest. So that actually flows really nicely into the next thing I wanted to ask you about, which is, you know, Anton and I are both, I think what you would safely call AI skeptics. And Anton and I spend a lot of time looking at the marketing in the security industry and we're often quite skeptical about it. We see companies saying, oh, bad guys are using AI, you should use AI. Have you seen the bad guys you're defending Google against? Use AI? Is that a thing that's happening out there you're aware of?

We haven't seen this at scale in a way that really concerns us. We certainly know that it's possible. I think part of the reason might be that they haven't had a need for that yet.

Hmm. Say more about what you mean there.

Well, if you can be successful making money without AI, why would you bother? Because it's really complicated.

Got it. So you're saying that when the bad guys get together every year for their conference at the Black Sea, They're not actually putting together a talk based on their techniques, they're just bragging about how many new cars they bought.

Potentially, but I'll put it a different way. It's like, why use AI when it's just frankly cheaper to hire people to write? You can use AI of all everyone you hire to do your work for you. So let's say, where might AI be useful? It would be maybe in the targeted phishing emails that you generate about individuals, right? And at present, it would take you a lot of work to craft an AI engine to make convincing phishing lures that were targeted toward individuals? Isn't it just cheaper to hire a legion of people to do it? And they have like a brain, which is essentially a form of AI, a lot less expensive.

Yeah.

At least on a fixed cost basis.

I believe that. Okay. Okay. I like that. That's a good answer. So I guess then, have you seen adversaries not using AI, but taking techniques that disrupt our ability to use our defensive AI? Or are they at least aware of the AI that's going on?

Absolutely. They know that we have AI and they are very clever at creating techniques to get around it. They know that AI can sometimes be brittle and they actually set up systems to test. So it's quite clear to us that they are probing our defenses before doing their attacks.

Hmm. That makes a lot of sense. And are there other understandable examples of how it's brittle that you could share without telling the adversaries where we're brittle?

I think one that's fairly obvious, using alternate character sets. So these are called homoglyphs. And so with humans, you can make Google look like Google, but it's actually a different set of words. And they assume that we have different classifiers based on specific word fragments rather than what the word visually looks like.

Ah.

You know, so, you know, Google with like the O's being from some other alphabet. So that's a very brutal thing where it's like, oh, well, I'll just use a different character set. Looks like an O when it's rendered, you know, the computer thinks it's something else and Totally.

That makes a ton of sense.

You know, we're on to that. You know, that's something we've addressed. And, you know, part of that is, you know, how might we think about processing emails like a human would and getting as close to that as possible?

That's really interesting. We should actually connect outside of the podcast on the homoglyph attack stuff. There's interesting potential collaboration on that. Moving right along from that, you know, there's like so much energy around AI in the industry, and there's a lot of, you know, PMs that are starting out or TLs that are starting out. who want to get into AI, who are in the security space. Do you have advice other than don't to a PM or a TL that wanted to get into using AI to solve these kinds of problems?

Well, I'd also repeat back, don't do that. But after we get through this for a bit, the advice I really have is around think of this as a process and a journey. It's not an end state. There's a well-known data scientist once gave me the slogan, a worthy problem is one that is worth solving multiple times.

What does that mean?

That means that it is a very challenging problem that's very meaningful and that there won't be just one solution to fix it. You'll want to iterate on that and do better on that problem over time. And what that means applied to machine learning is start as simple as is useful to your users. Don't start for the simple just for its own case, but let's say you could move the needle on your system's performance by 10% and you determine that's enough. Start there. Do that as quickly as possible and iterate. Add more features. Add a different algorithm and make sure you set a metric that correlates closely to what success in your product looks like. And then essentially rapidly iterate on that. And you will learn so much and you'll accomplish so much and you'll actually have a ton of fun at it because you're watching this number go up over time. So that's really my advice when you're approaching machine learning, understand your problem.

Mm-hmm.

Get a metric that you think you can optimize on and just get to work and iterate on that with the simplest things first.

Huh. That advice echoes a lot of the other conversations we've had on the podcast, particularly around the challenge of rolling out zero trust in organizations. The advice is to start small, have an achievable goal, and then iterate.

Absolutely. I mean, I guess that's very common in the engineering space, right? It's launch and iterate. That holds true. But I think the thing that undergirds is useful and doesn't overly damage your users.

That makes a ton of sense. Doesn't damage your users. So on that one, are there cases where we've tried to use ML and we damaged our users? Have we stubbed people's toes and skinned their knees with AI in the past?

Well, I would like to say no, and sometimes you don't even know, like, all the different places. But one thing we're very careful about, let's say we're applying IA to phishing or to malware, is what we call our false positive rate. If we tag something as phishing but it's not and they never see it, that's actually a live interaction that might have been important to them. And so It's unfortunate but true that we will deploy models that don't perform to our expectations and we need to kind of revert. And here I think there needs to be humility to assume that if you launch something, you need to have your metrics in place to identify that failure in advance. Most of us launch something expected to work perfectly. I think you almost want to have the inverse and prepare yourself for failure. Launch something, assume it's not going to work, and then be pleased if it does work.

Uh-huh.

So it's almost a mindset shift because It's quite common that what worked well in theory and in your batch testing or your sampling doesn't work out in real life.

So should users then, or, you know, these PMs and TLs out there listening to the podcast, also keep in mind a, like, downside metric and keep an eye on that as well?

Absolutely. I mean, I think they almost go in parallel. You need your target metric and what I call your guardrail metric. So maybe you're trying to catch a certain type of bad activity, you know, that's your recall, but then you need your false positive metric Right in line and define in advance how much damage you're willing to sustain before you roll back.

That makes a ton of sense. So with that, I think we'll come up really on kind of our traditional closing questions. First, do you have a practical tip for people embarking on this journey, which we've kind of just talked about, but like one really succinct tip for people who are looking at an AI journey or thinking about using AI for a security system other than don't, which is what Anton and I are going to say?

It's all about framing. Are you setting your model up for success, and how are you addressing the failure?

Okay, think about the benefit and the risk and be ready for failure.

Actually, oftentimes with machine learning, your ultimate successful outcome is in gracefully handling the failure of the model.

What do you mean by that?

Well, let's use Gmail as an example. The spam folder is a great kind of like mechanism to accommodate failure. of a machine learning model because for the most part everyone's very pleased when you don't see spam in your inbox until they find out that they never got an email. So what if like Gmail or email never had a spam folder and it just nuked your emails? That kind of would be a bad outcome though, right?

That would be.

At least now we have a graceful way to degrade.

Okay, so the spam filter is graceful failure. I like that.

Yeah, it's a really good example of graceful failure. So maybe in synopsis is like If you want to start off in machine learning, come up with really good graceful failure. If you can, then you have an opportunity to allow your model to grow into a successful adult.

I really like this because we get to give people advice about spam filters and spam folders. I usually treat marketing emails that contain AI as something that belongs in the spam folder. So this is all really fitting together. The final question that we like to close with is, do you have recommended reading other than, of course, checking out your talk with Neil from the Security Sessions There's 2 AI podcasts I do.

One is called the TWIML AI Podcast, and I think that's more of an in-depth practical view. And then there's an Intel AI podcast I like for in terms of AI applied to real-world applications.

Oh, great. Well then, listeners, we will get those into the show notes and you can click on them from your podcasting app of choice. And in addition to subscribing to this show, you can subscribe to those shows as well. Andy, thank you so much for joining us today. Really fun to talk about AI. I know we didn't talk as much maybe about your talk itself, but I think where we went talking about AI was both practical and, for me anyway, I learned a lot about how you think about the problem. So thank you so much for joining us.

Thanks, Tim. Thanks for having us here. Appreciate it.

And now we are at time. Thanks for listening and of course for subscribing as well. Hopefully it was a fun one. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or many other podcast apps, pretty much wherever you get your podcasts. Now, you can also find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. It's pretty easy to remember, right? cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can also follow us on Twitter, twitter.com/cloudsecpodcast. And of course, your hosts are on Twitter as well, @anton_chuvakin and @TimPeacock. I'm sorry, I still can't get used to this whole leading_team. Tweet at us, email us, argue with us. And if you like or hate what you hear, we can invite you to the next episode. See you on the next Cloud Security Podcast. And by the way, quick disclaimer, no ML unicorns were harmed in the production of this episode. Security.