EP50 The Epic Battle: Machine Learning vs Millions of Malicious Documents

Security. Hi there, welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the product manager for threat detection here in cloud, and Anton Chuvakin, a reformed analyst and esteemed member of the cloud security team here at Google. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you like our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show and argue with your hosts on Twitter as well, twitter.com/cloudsecpodcast. Anton, we've got Eli Bernstein joining us again today, and this is an exciting episode because we're talking about a very, very real application of machine learning to the problem of threat detection. Which I'm always the first person, I think, after you to say, you can't do that.

I would say it wasn't I can't do that. It was more like most people who say that they did it really didn't, or most people who really did do it didn't succeed, or they achieved much less but marketed it much more. It's less about it can't work. It's more about it kind of hasn't so far despite all the noise. But you know what? We're going to have an example today where it very clearly did work.

That's what's fun about this. You know, it's not like we're doing this one out of marketing. We've actually been doing this for a long time and it's not a service we sell. We're talking about this just because we think it's an interesting thing.

Yes, correct. And it's also, it will become very clear from the episode that some of the stuff is kind of magical, but it's also magical as in it works well with other techniques. It works well together with rules. It works well together with very non-magical stuff that's just hard work.

Yes.

So to me, this episode would prove to skeptics that AI is no good in security. It would prove to optimists that AI is great in security.

I think the truth, as always, might lay somewhere in between. And with that, let's welcome today's guest, Elie Berstein from Google.

Elie, this episode draws on a talk available in the podcast materials. I'm assuming some of our listeners didn't do their homework. So could you summarize the gist of your talk for the audience so they don't have to pause the episode, go read the thing, and then come back to us?

Absolutely. Thank you for having me, Tim.

Of course.

So the talk you're referring to is a talk I did, I think, 2 or 3 years ago around the challenges when you try to apply machine learning for anti-abuse in practice, based on our experience at Google. And we try to go over the Main pitfalls, not necessarily having answers for all of them, of course, as we're going to discuss, lots of open questions, lots of things to figure out. But we try to be very concrete around what happens when you have a false positive. How do you deal with people actively trying to tamper with your neural networks or your classifier, as we see regularly, and things like this. So I think we'll go about some of those questions, right? So that's the gist of the talk. As usual, The talk is available after the fact. If you don't do your homework, maybe you want, you're inspired to look at it after that. I'm sure there will be a link somewhere.

There is.

Awesome.

When I was reading the talk, one thing that struck me kind of hard is that in one place you mentioned processing millions of documents in milliseconds. And frankly, I don't even know if my brain can parse this, like millions of documents in milliseconds. What is involved in making this work? People may not get the scale of what you're doing. So like, How do you process millions of documents in milliseconds? How does it work?

Difficultly. I think the 100 milliseconds, you know, came from the latency we need to have to scan documents that flow through Gmail or Google Drive, right? At that point, a lot of the content we have needs to go through a service, right? The service is the attachment scanner, right, that Google has. that is also used for Google Workspace. So it's used for end users and for companies as well, right? If we were to spend, I don't know, like 10 minutes per document, people would not receive their email. You would upload or you want to download a Drive file and then you don't get your Drive file. So there's a requirement production constraint. And this is something that production teams are running, right? At Google, we are research teams. We help building kind of the machine learning side and the static analyzer and stuff like that. The production team is running that at scale. And I think it's beyond the 100 milliseconds. It has to be, if you lift a little bit the COVID right? We have to do it worldwide. We have to do it 24/7 reliably across the world for every document flowing through Gmail or every attachment flowing through Gmail to make sure none of them contain malware with all the difficulties we talked about. This notion of adversarial attack, we can briefly touch upon. It's 100 milliseconds. It's very, very difficult to do it with a lot of parallel computation and containers, as always. That's Google scale, right? That's what it amounts to today to be able to scan and protect users from malware.

Essentially, you do it because we have Google scale, because we are Google. Short and somewhat non-techie answer is that if you have to process documents in 100 milliseconds, then we make it so because we are Google and we have the infrastructure. Is it too snide of a remark or?

A little bit too snide.

No, Anton, I think what you've done is you've rederive the expression, necessity is the mother of invention. Elie had to do this, so he did it.

Kind of. I think we spend a lot of effort. It's not only Google scale, you know, CPUs, and you can scan each document in parallel, right? That's what we do. Every document we have, I don't know how many thousands of parallel computations where the service absorbs the documents. There is still a lot of work which is done by the anti-abuse teams to Sandbox the various containers to make sure that, well, if there is malware, you don't go into process something which is dangerous without limiting its access, making sure that all the library and function calls are as fast as we can using caching. So there's a lot of engineering behind the scenes. I do think other companies do that very well too, right? We're not the only one who can do it, I don't think. But you're right, it requires a lot of engineering. It requires a lot of ongoing effort. You might be able to process 95% of the document that fast. And then the 5 last percent, the 5, 1%, the last 1%, which are the difficult cases, is disproportionately where you put a lot of effort, right? It's kind of this Pareto law. You can get 80% there. And then the last 20%, you spend 10 years, 20 years trying to perfect your technique so that it actually works well. That's probably what makes it unique to be like a large company and have enough resources to do it. And that's the very, very expensive cost of security in general. It's the ongoing battle and the ongoing effort to optimize. And the more you do, the more you process advanced threat, the more you actually cram into detection there, right? What we were doing 10 years ago and what we do now, the amount of complexity have increased drastically, but we still have such a little envelope of time. So I don't know what goes faster. Is it the complexity or is it the amount of power computation? And Moore's Law is dead. So CPU start to slow down. So now you have to use TPUs and you have to do like different type of accelerator. So it's a whole stack effort. And we really do benefit for sure of the advance that Google is bringing in terms of computation efficiency, accelerators, to be able to do more in the same amount of time. I don't know how long it's going to be sustainable, but so far we've made good progress.

That's pretty clear. And I think I apologize for my kind of like, hey, just throw Google scale and it's solved. I understand that I used to joke that even a file copy at petabyte scale is a major engineering effort that you require— may require a PhD to accomplish the task. But I want to switch gears to the other big complexity of this, is that a lot of vendors, a lot of people throw ML at various security problems, and occasionally the results are quite hilarious, or they're just bad. We know now that malicious documents is a good problem for ML, like, because we made it work. So do you know what makes this security problem a good candidate for applying machine learning? Could we have used just rules? And if not, what would we miss? Why is ML Unicorn doing it this time?

So we're doing the hard question today. I see. Okay. Let's do the hard question today.

Oh no, this is the starting of the hard questions.

They get worse from now on.

Oh, this is just the starting.

Oh boy.

Okay. Well, what did make it a good candidate? The first thing to say is, at the time we did the talk, which was 2 years ago, it was already 4 years in the making. So it didn't happen overnight. We did test a lot of things. The reason why we did that is, I think, twofold. The first one is we've been driven by data. We're data-driven. As a reminder, Gmail does not authorize executables as attachments. So when we started to work on how can we improve the safety of Gmail users, whether they are enterprise or they are end users, We had only attachment to work with. That constrained already the, I would say, the space. If you think about, we could have started with Drive, and Drive do authorize attachment of binary and so forth. However, when we look at the data, we felt the telemetry was showcasing a very, very strong rise into Office documents and PDFs, and Office in general, like document as a precursor of the malware. So for the people who are joining us, what usually happens in a kill chain of malware is the attacker will send you a document which will either execute a macro, sometimes an exploit, although it's rare these days, and then you click on it, and then it downloads some sort of a binary, decrypts it potentially, and then you execute it. That is the end payload. So you have the precursor as a document, and you have the payload, which is more along the line of the binary itself. So we don't have the payload for Gmail. So that was one of the things. Now what we know is from the machine learning standpoint, a lot of progress has been made since 2015 on natural language processing, which is essentially processing text. Documents have, contrary to binary, which are compiled, have script and have content, which is mostly, if you really abstract it away, very, very close to programming languages. It's not really the case, and there is a lot of transformation to do, but You can apply a lot of programming languages techniques on it, and you can apply a lot of natural language processing ideas from the machine learning side of that to the field. So we thought we'd give it a try, and we found the right formula. Just to be clear, the machine learning is only applied to very, very specific threats, including VBA-based and also Excel 4-based detection, which are actually the ones who have very complicated scripts. And in that specific case, it works. So I think part of the success was driven by, as I said, necessities. The other one was driven by opportunity of benefiting from a lot of the work done in other teams at Google who do a lot of research on AI, who had a lot of good ideas. And when you bring those two and you try to make it efficient, it kind of works. I think it boils back to also a business need, right? Kind of a need. It's in the talk, right? I try to make a chart on complexity of the processing versus the time it takes. On one hand, you have antivirus, which are extremely fast, and on the other hand, are missing a bunch of zero-day threats, right? A variation of attacks, or Excel 441 was sometimes undetected, stuff like that. And on the extreme end, we have a sandbox where you open the document and you see what happens, and you instrument like a full VM. At 100 milliseconds, we can't really do this. So we can do that for very, very specific advanced protection reasons and for very, very suspicious documents. But in general, we cannot do it at scale. So it seemed that you could pay a little bit more than an AV, but not too much to get way more in depth. And I think machine learning is not— we didn't try to replace everything. I think a lot of the startups which have kind of interesting results, I would say, is because they try to do everything at once, where we were trying to say, okay, we try to fit the technology in between and add it to our tool belt to increase resilience. More often than not, we are mostly concerned about the lowest detection rate. We're not concerned about the highest detection rate. You know, it's like, are you at 99.9% at peak? Not really what's important. What's important is how low do we dip when there is an active attack. Over the last few days, there is, for example, Emotet, which is hammering the world. If you follow the press, there is a lot of massive campaigns these days, mostly driven by Office documents and Excel again. I mean, I can't really say the official numbers, but let's say something which has contributed significantly to stop things which have not been seen by any other technology was deep learning because it extrapolates. But again, it misses things that other defenses have detected. And I think that's what makes it very, very different. And that's what we try to convey in the talk is we apply AI as a belt and suspender idea rather than just making it as a standalone. And I think it will have some pros on certain aspects and it will have some deficiencies on others. And we really, really aim for resilience. And I think that's what makes security a very, very different field is we want to have as much resilience and as much as kind of double security or security in depth, right? That's the main principle that a lot of people mentioned. So that's the idea. I think that's what makes it different is we took a very, very specific problem. And really try to add to the thing, not to replace. I don't know if that makes sense.

No, that makes a ton of sense. I want to keep getting deep on the ML piece here because I found one of the techniques here really interesting. One of the things that you described doing is reanalyzing old samples as a way of finding ground truth. Can you first off explain to users what ground truth is, why it's important, and then in the discipline of ML-based detection engineering, what you're really trying to do here?

3 questions?

Yeah.

Okay.

I'll try to answer all 3.

Perfect.

Let's start with the first one. What is ground truth? When you train machine learning, machine learning learns by example. So you hope to be able to teach the machine what to recognize. And the power of the machine is not really to invent new things or to think. What it does really well, it interpolates. Interpolate means if there are 2 samples and you have a slight variation between the 2, it will be able to say, well, that is something along those lines. in a very, very high-dimensional, complicated function approximation, but that's what it does. It tries to interpolate extremely well, so it deals really, really well with variations, provided you train correctly. So in reality, when people talk about new malware, there's not a whole lot of new, new things.

Rumsfeld, get out of here.

Yeah, exactly. And I think by constraint, what you could do with VBA or Excel or PDF is kind of well-known. There may be zero days, right? And zero days will completely catch you off guard. And I do think the machine learning will not see them, to be clear. So that's why you need other technology. But for most of the things they do, they do a flavor of this, a flavor of that. Last year, it was in the press. Many groups, attack groups, because VBA was so well defended, started to use Excel 4. Excel 4 is a 20-year-old format. So when we talk about looking into the past, Some of the attacks are using very, very old legacy languages that were invented 20 years ago. So you get this idea of why looking to the past is important. The other problem we have is when you train, you need to have an example of what you want to see, but you also need to have what we call a verdict. And the verdict is essentially whether this document is malicious or not. That is very, very hard to get if you don't wait a little bit because AVs, us, any kind of company improve their scanners and we discover what was missed. Sometimes we have user reports, we have industry reports. And so as you adjust your scanner, we relabel with current knowledge something which was done a few days back. And so basically we improve our understanding of the past. That's what I meant by looking in the past is you can't tell if with a document today the machine learning can make a guess. And what might happen in some of the cases is every other system says it's not malware and the machine learning says, no, no, I'm sure. So now you have kind of this ambiguity of who to trust. Turns out that based now on 2 years of looking hindsight, the machine learning is mostly right. Mostly it's like 99%. I don't want to put the exact number because I'm not sure exactly what it is, but over 99% sure. And that when that happens, we tend to make the decision to spam folder Or to block the document because the machine learning has seen something or interpolated into a way. That said, when we retrain the new model, and the model has to be retrained because the landscape keeps changing. And that's one of the things which is different from normal machine learning. If you try to recognize a dog or a cat during nuclear winter, your cat is going to look the same. The problem is when we look at the type of malware, they are different, right? They are 60 to 80% different day to day. As I said, not completely out of the blue new thing. It's just variation and constant change by the attacker who tries to evade systems. And they probably have a copy of every AV they have. There are even underground services which allow you to test your malicious payload against all the known AV in a private version. They call that private VT sometimes. It's very, very funny.

Yeah.

So some stuff like that, right?

This is what I find so fascinating about using ML for security problems. You don't have geese out there trying to look like ducks for the most part, but we do have an active adversary that we're playing against here. Somebody interested in buying Toyotas isn't pretending to look like they're interested in buying Fords. Advertising has a much easier problem than we have in security when it comes to applying ML to these problems.

I don't know.

You disagree?

I don't know that.

But we do have vendors who look like complete ducks when they try using ML naively.

I don't know about that either. I do say, you know, ads have their own problem. It's very, very hard for them, especially when you have online ads. It's a little bit of a digression, but they also have this problem of people trying to make fake ads, ads which are not legitimate and things like that. So there is a lot of difficulty to not have ads serving malware, spreading disinformation, spreading, redirecting you to counterfeits. There is quite a bit of challenges on that. I would say recognizing cat versus dog is a pretty safe bet or something you can do, which would probably work okay. And yes, we don't have that in security. But in certain cases, as I said, for, in particular for documents, it's very constrained what are the primitives underneath it. So with that, you can train the machine learning, which has a pretty good way to interpolate. I'm very careful to not say detect new things or zero days, as I'm unsure that it's doable. I'm unsure it is doing it. Maybe it does. Obviously, it's worth reiterating, we did not look at our user content. So we can also not have a lot of certainty around most of the documents. Documents we can look at are either reported to us or they might show up later on, onto public sources, but we don't inspect public email content. So sometimes, you know, we detect and then We only have statistical estimates. So if there is a zero-day, maybe it was blocked, but I have no way to say that. What we could say for sure is we have seen the machine learning catch part of large campaigns and add sometimes up to 100% increased detection, up to 200%, where it means like 2/3 were missed by other techniques because it is able to interpolate better. So I think that that's what it does. It's a very underwhelming thing, by the way. If you think about it, it's very, very far from the promise of a robot killing all humans. Fixing all problems.

Yes, yes, that's right. The replacement activities, like you don't need rules anymore, just use our ML classifier and it always gives you the right answer. It's not like we are far from that. We're just not going there.

I think some people try. There is this field called certified ML where they try to tell you that they have some guarantees that the ML will not be too far off. I do think for a stable problem, that might be doable. Now, what is a stable problem? Those are fairly constrained. That's not going to be a self-driving car. Self-driving car is not a stable problem because there are so many things which can happen on the road. That's not going to be one of those. A stable problem might be controlling an elevator or maybe processing—

Almost nothing in cyber, basically. In our beloved domain, almost nothing is like that.

In our domain, it is extremely unlikely that we'll find something where we have strong guarantees of performances. But I've never seen also any vendor guarantee detection rate, even traditional AVs or even Google. You can't promise under adversarial attacks that you're not going to be evaded. You could certainly do everything you can to mitigate that and have control in place and have multiple layers of defense, but there's always something that could go through.

Which kind of reminds me of the other angle on this about stable problems. Do we have any evidence that attackers are trying to evolve and use anything that's counter-ML or anything that tries to attack/disrupt or bypass ML techniques specifically? Obviously, people try to bypass rules by doing things differently. In your work, do you see any evidence of the attackers Countering the ML techniques in particular?

That's what you meant by we were just starting a few minutes ago on the hard questions.

With the easy questions.

Yes, correct.

That's right.

That is, now we are getting to the hard questions.

Okay. The reason why it's a hard question is I don't have an answer. However, I can tell you 2 things. One, I think there was a report, maybe by Gartner, who said they think 60% of the attacks they see are anti-ML. I'm not sure how they get to that estimate, so we should We do see people trying to game the system very aggressively. We don't have, as far as I can tell at Google, any system which is only ML-based. So I don't know if the attacker is able to literally optimize against one specific component. I would say they do optimize very aggressively against what we do. As I mentioned, what's very, very interesting is a few months after we launched our ML detector for VBA, we saw a very, very strong shift of how the attackers were trying to evade detection by moving to Excel 4. It was already a shift, but we saw a very, very large shift, say like 30% or something like that, I think, on the amount of VBA decreased significantly. The amount of Excel 4 did increase significantly because they realized it was not something we had ML for at that point.

Hmm.

So there is Definitely some sort of reaction we can see. So we can see them trying to evade for sure. We see them trying to poison our dataset, people doing false reporting on Gmail, spam foldering things, removing things out of the spam folder is a classic for many, many years. So I do think there are both of those. We do see some of those. And then the last thing, which is a little bit more depressing, if you take a little bit of the scientific hat, What we're looking here is called out-of-dataset, right? So out-of-dataset sample. There is a lot of negative results. We think it's as hard to detect them as to know whether or not this is something you haven't seen. So it's out-of-distribution sample, ODS. It's really, really hard to detect. And I think some people argue in the ML community that you can't do it reliably. You can maybe detect some specific technique and some evasion technique if you are looking for them. But that's supposing you know what the attacker is using to do that. So if one day there is like an off-the-shelf tool to do this, you could probably optimize that. I think that's part of, people do some stuff like this for deepfakes, where they try to look for known libraries who generate deepfakes and try to detect those. But you know what you're looking for. If you have an attacker and you don't really know what they're doing, it will be really, really hard. So I don't even think we can detect them.

That's nice.

I think that's the short answer. The short answer is I think it's really, really hard to detect. And there is no theoretical foundation to say it's actually doable. I think you can find some artifacts if you have seen multiple times the same technique used, and we might be able to detect some of it. Now, how much we have, we don't measure that. The closest I've ever seen was a Gartner report, if I remember it correctly.

Well, those Gartner reports, you know what we always say about them around here.

Oh, we thank them. They are great people. They are super nice. They sometimes reach out to ask questions. They are super nice people. So I like Gartner.

They're very nice people and they intend to do a good thing in the world. I want to move us towards wrapping up because we're just about at time here. Our traditional closing questions are maybe not super applicable given how on the edge the work is that you're doing, but I'll ask it anyway. Do you have one weird tip for people to get better at using ML for security? And what's your recommended follow-up reading aside from also the talk that we will again link to in the show notes?

One tip to get better at cybersecurity?

Specifically using ML.

Using ML for security?

Please don't say get a petabyte of documents because this would be like cheating.

Okay. Actually, I was not thinking of that at all. I would say pick one simple thing and do it yourself and play with it. Just getting the thing working end-to-end will give you a lot of insight into how the thing works and what are the difficulties. It's probably less hard than people might think to get started. You get to 80% of the result. So really, really hard part of ML is the last 20%. You're getting from 85% to 90%. That's the place where it gets very, very hard. But simply picking up one idea, I want to detect phishing pages, I want to reduce— there was a very, very nice dataset by Sophos on malware to do detection. Pick any of those, apply the basic deep learning techniques, try to get it to work. A, it's going to be very satisfying because you get something to work. You get your own detector yourself. You should be happy. And B, you will see all the different types of difficulty from getting the data to curating the data to training to getting the GPU to work to do deep learning, which is not that easy actually. And then get through all those stages and you get one experience and this experience will make it clear all the things you could learn and all the directions you can go. I would say just take it very, very narrow. I think when people look at ML, they tend to look at courses which have a lot of theory and have a lot of application. And you do a little bit of that, a little bit of GAN, a little bit of vision, a little bit of NLP. And that seems impossible and you don't know where to start. And I say, that's okay. Pick one. Pick the simplest neural network. The simplest neural network will get you somewhere. It's not going to be state of the art, but that's okay. That's not the goal. You get a reasonable network on a reasonable problem will work reasonably well. And I think AI is to the point now where you can do that. You can do that at home. You can use Colab if you don't have a GPU, which is the online service Google provides for free to train models that is heavily used. You train your own model, you have the satisfaction to see it climb, you see your accuracy, you play. Its core is a very, very hands-on practice. It's very, very practical. It's very, very much through experimentation that you learn about it. So I encourage a lot of experimentation. I would actually steer away from theory first and just try to get a sense of what works. And then the theory will make more sense when you get it to work. So completely the reverse advice that I hear a lot.

No, I love this.

For an episode where so much of this was, well, this is really hard, this is a very open space, this is very difficult to get right, your advice is very practical. It's just go try it, go do it, and go do it in a way that is achievable, narrowly focused, and listeners could actually hang up this episode and go do.

It's amazing.

As I said, I think we are to the point today where with the tooling which exists and the resources which exist, you can, if you put your mind to it, get to 80% of the efficiency. I think everyone can get to 80% if you put your mind to it. It's not like very, very— you can get there. So 20 last percent, and that's what turns the difference between a prototype that you do for learning to an actual product. is really, really hard. And then what distinguishes a great product from a good product in AI is when you get to the last 1%, right? And this last 1% requires even more effort and way more advanced and way more computation than everything else. It has these kind of exponential difficulties and then you have to maintain it. So I do think you can get started very easily. Making it work in practice is a whole different level of skills. And maintaining it and keeping it days in and days out under extreme adversarial context is where it's really, really hard. And that's why there is so few AV company really, right? And really good company. They put a lot of resources, it's very expensive for them to do it because it's really, really hard. So encourage you, get to the A-tier and then you'll get a lot of fun at it and it will be awesome.

That's a fantastic answer. Elie, thank you so much for joining us today.

Of course.

Bye-bye.

Thank you so much. Thank you.

And now we are at time. Thank you very much for listening and of course for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, no more website jokes, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. And of course, your hosts are also on Twitter, pretty active, I would say. Anton underscore Chuvakin and underscore Tim Peacock. Tweet at us, email us, argue with us, and if we like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode, and of course, welcome to 2022.

Security.