EP124 Safe Browsing: Lessons from How Google Secures Five Billion Devices at Low False Positive Rates

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday Pacific time, please do hit your subscribe button. You can rate your podcast if that's an option in your app and you'd feel comfortable giving us a good review. You can follow the show and argue with your hosts on LinkedIn as well. We've got quite a bit of commenting on every episode. Join the discussion. It's a lot of fun. Anton, we are talking about a pretty fun multi-billion, not user, but But device product today.

Yes.

And moreover, while Google is very proud of having multiple products that are used by billions of users, we only have very few security products that cover billions of devices. And to me, the scale of this is staggering. Sometimes you hear of some security technology that is adopted by 1,000 companies, and you think, wow, that's wide adoption. But here's a piece of tech that's adopted to cover billions of devices. Billions with a B.

That's a big number.

It's a big number.

So I thought this was a pretty interesting episode, not only for the scale, but because Panos told a story about using ML a long time ago in a pretty interesting way. And he's the first guest to come on the show and tell us a story about breaking google.com. So maybe with those 2 teasers, listeners, let's welcome today's guest. Today we are joined by Panos Mavromatis, an engineering director at Google Cloud Security. Panos, delighted to have you today. And you're going to be talking about one of my favorite things at Google, one of our billion-user products, Safe Browsing. I'm sure a lot of our listeners have heard of Safe Browsing, but could you give us, since you started it, the 30-second overview and how it got started?

Excited to be here, too. Thank you, team. Thank you, Anton, for hosting. Safe Browsing is Google's, I would say, core threat protection product. In particular, it protects against malware, phishing, and other social engineering attacks on the web. It has been built into Chrome but also serves many of the other browsers. It's also part of Google's internal threat protection stack, which includes technology built by other teams outside Google. It protects Google's users on products such as Gmail, Search, YouTube, et cetera. As you said, through the coverage it has with browsers, it protects over 5 billion devices today.

That's— we should probably pause for a moment and, like—

Did you say that number again?

Yeah, I was like, I'm going to interrupt him and say, 5 billion devices, right, on this planet?

Correct.

Now—

That's a lot of devices.

We don't say users because we know, we recognize some users have multiple devices. But yes, we're proud to provide that coverage.

That's amazing.

That's scale. That is scale.

Wow.

OK, so how did this get started? How did you get to 5 billion devices? Surely not overnight.

Not overnight. It started before I joined Google. So I want to correct you there that I—

Oh, listeners, I was wrong already. We just got started, and already I was wrong. This is great.

I'm not the founder of Safe Browsing per se, but I'm probably the longest-running member, or the longest-lasting member, if you want. It started in late 2005. And it was a Google Labs beta product, and it was an anti-phishing plugin for Firefox. If you can remember back to 2005, there was no Chrome, no Android, I think, no iPhone certainly, lots and lots of Internet Explorer.

A nervous chuckle about that.

Exactly. Firefox was our first vehicle of choice. I joined in 2006, straight out of college, so young and naive and super excited to come solve phishing. I thought I would join a team and learn how to code and learn about scale at Google. And I would, as soon as I would be done with those things, I would move on, right? But almost 17 years later, I'm a little bit less optimistic that we can, quote unquote, solve these problems.

Yeah.

I joined and I started working on exploit detection on the web. Our goal was to scan as much of the search index as possible, and we built a scanner that looked for mostly hacked sites that distributed exploit kits. And we talked about Internet Explorer. At that time, both the browser and the plugins in the browser, things like Flash and QuickTime and Java, you know.

Oh my God, this is really a long time ago.

It is a long time ago.

Do you remember applets?

Correct, yeah.

ActiveX and, you know, lots and lots of software, right, that was basically running in your browser there exposed, if you want, right? Taking content from the internet and boom, running it, right? No sandboxing generally. So lots and lots of exploits were written for Not really zero days, but really just months-long or year-long known vulnerabilities. And this ecosystem of browser plugins was never built in a way of auto-updating or keeping patching, et cetera. So what would happen is you would visit a website, and if the website was hacked without any visible action, without any requirement to click on anything, your computer would get infected. Maybe you would see more and more toolbars appearing in your browser as a result. All of this was before Chrome, before Android, as I mentioned. Once those platforms came online, Safe Browsing team expanded to scan for malicious Chrome extensions, malicious Android apps. We also expanded on going deeper with phishing protections, malware downloads that don't rely on exploits, But rely on social engineering, basically tricking the user to believe that they should manually run and download and run a piece of software.

The scale of the main system had to grow, right? You had to expand the coverage for simply more websites, right?

Yes. Even in the beginning, we had trouble scanning every single website with our most sort of expensive scanner. So what we built was a— we had 2 layers of scanners, if you want. The first was a virtual machine that would load a website in a browser and wait and see if it got infected.

That involves a massive amount of money to run it for every website on the planet.

Yes.

So we were lucky that Google had a lot of servers lying around in order to be able to scale for spikes, etc. So we actually used resources that were unused. We didn't go purchase new machines, but we certainly went and made those— utilized those machines as much as we could. The other thing we had to do was to put a faster scanner in front of that, something that could actually truly scale to the billions of websites that Google had in the index, even in 2006. And what we did there was we looked at the content that Google had already crawled. And I remember I started with writing some simple heuristics like, oh, can I find a bunch of iframes appended to the end of the page? This was the sort of preferred MO for the attackers. They would hack the website. They wouldn't drop the actual exploit kit in the website. They would drop an iframe that pointed to— Oh, I see. their own server, and then they could, at that point, they could serve their own exploit kit from their server. Some of these websites were so vulnerable that they kept getting hacked multiple times. So you would sometimes see a website with 20 different iframes appended at the end of it.

Oh my God.

Yeah, these are the times. Yes. But let's remember, this is the time before crypto miners, so they couldn't monetize all that well. At the time, which is kind of almost like a missed opportunity, I guess. But it's a 10-year— actually, 12 years before crypto miners were invented. Yeah. Wow.

Yeah, that's right. That's right. Then we also realized that we could also use the output of the more expensive VM-based system as ground truth and train a model, figure out which of these websites that Google has crawled are more likely to lead to an exploit kit detection if we do put it through the VM. So that was around 2008 when we built our first ML model to leverage the system, and it was fully and automatically trained by the second stage.

That actually makes sense. And I think a lot of this tech, a lot of the approaches are probably used by other vendors years after us. In that sense, this type of stuff is was cool back in late 2010s, but we did it in the late 2000s, right? So like way before this.

That's right.

So I want to turn this a little bit in a different direction. And of course, initially, the focus was to make internet more secure, and it was essentially a consumer product. But today, we sort of treat Safe Browsing as both a consumer-focused and business-focused product, which raises to me the question of the threat model. So is there any thinking behind it? Like, now that Safe Browsing helps consumers and businesses, how are we dealing with the threat models?

Threats are quite similar, I would say, perhaps surprisingly.

That is surprising.

It is surprising.

Anecdotally, I think there's a big overlap between IT admins and gamers, for example, and a large overlap Large portion of malware downloads we see these days are tricking users into believing that they are game hacks, and they even tell them, "Hey, look! If you have an endpoint security or antivirus product or safe browsing, they will tell you that this is a bad download. But don't believe it, right? It's just that they don't want you to hack your game, right?" A lot of these pages say, you know, turn off safe browsing, turn off AV. And we do, I think, these days, a lot of employees, contractors, they infect their corp laptops with the same kind of malware that they infect their home computers. And sometimes those things are exactly the same machine. Likewise, on password phishing, while a lot of businesses have adopted the latest and greatest technology, many businesses are stuck with really old systems. And old software that requires a password, some of them support a second factor like an OTP, but some systems don't even support an OTP. And even as an SMS OTP, for example, that is still fishable. If someone has convinced you to type your password into a page, then it's not that hard to also convince you to type the OTP that you got. On that page, then the OTP.

Of course, you type one, you type the other.

Yes, like you've already been fooled to type your password, right? So what OTP does is prevent you know massive credential stuffing, of course, right? But it doesn't prevent phishing.

He said credential stuffing, by the way. Ah, Tim, are you supposed to react to this? Right?

It's my favorite. Panos, you know I worked at the company that invented that term, right?

But were you the— would you say you were the inventor, therefore?

Just as you were not the inventor of safe browsing, I was not the inventor of that term. But I have right here on my desk a little No Bots pin from that company. It still sits on my desk to this day.

Yeah. Certainly, as I said, OTP credential stuffing should definitely be on all of our threat models. The thing about phishing, though, is that it does bypass. It can bypass. the basic credential stuffing protections.

Yeah.

And so one thing we do have we can take advantage of in the enterprise is that there is a security team. And so what Safe Browsing does, if you are in an enterprise environment, is we can give you visibility into what your workforce is doing when it comes to these types of security events. In the browser. What kind of files are they downloading? Which of those are potentially suspicious and malicious? Where are passwords being typed? What does Safe Browsing think about the reputation of the site on which passwords were typed? And even if you get that intelligence a little bit later, you can still react to it as a security admin. You can try to find the device and investigated. You can look for signs of account takeover or just force a reset of the password.

So we talked a little bit about making this work at scale by taking advantage of training machine learning models on websites that are likely to look sketchy. What else did we do to make this thing scale to be a 5 billion device product?

Yeah, I think as we expanded to scanning downloads, web pages for phishing and malware, We leverage similar principles. Number one, we try to classify everything using machines first and as quickly as possible. Now, there's things that pass through, so we do have, second principle, a set of humans who do review a small but important subset of, let's say, suspicious news sites or suspicious binaries.

Ah.

We can leverage our Telemetry to understand how to prioritize what to review. And number 3, we can leverage those human labels to train ML models so we can go back into the first system and have the automated detection get better and better in terms of automated classification.

So you considered carefully the feedback loops.

Correct.

But how did you make this scale to the whole internet? That's crazy.

Yeah, it's, I mean, I don't want like the whole picture of running every website in a VM. I know it was back then and not every website. Like, that blows my mind. But today we obviously use many, many other techniques, but we still run some websites in VMs to test for maliciousness, right?

We run websites, I would say these days, in a browser. And we also run a lot of binaries in VMs. Yes. It comes down to prioritization, as I said before. If we see a binary being new as well as being downloaded by a lot of users, then we would prioritize that. Then it comes to aggregation. If a website has thousands and thousands of URLs, pages on it, and all of them are being visited by users and a subset of them we see users typing a password in, we would prioritize those where a password entry is detected.

That makes sense. Okay, so there's smart ways of prioritizing and that helps you scale. That also makes sense.

So I wanted to switch to another subject possibly, namely that the system actually, it doesn't just work in the sense like a dancing bear. But it actually works well. And working well here often means false positives are low. We're not blocking people from going to their favorite websites. So, what is the magic for low false positives? I mean, there's an active discussion. Just earlier today, when we recorded, there was another spat on Twitter about people who argue about false positives. And this remains a contentious topic in the industry. And you achieve spectacularly low rates. So what is the magic behind the low false positive rates?

Yeah, the magic is probably a lot of hard work.

That is usually where magic comes from.

I already mentioned about our lists being pushed to about 5 billion devices. We also show warnings in Search. We give a signal to Gmail, which then decides to put things in spam folders. So it's a huge responsibility for— Safe browsing to keep our false positive rate as low as possible. Our first major incident was in 2009. We had flagged every search result on Google as potentially harmful.

Whoa.

For almost 45 minutes.

Whoa. That's a story to tell your children, right? Or on the podcast.

Yes.

How long did this go on for?

It went for about 45 minutes, I think.

Oh my God.

I mean, you can live without internet for 45 minutes back in the day, right?

My dad told me, ah, my computer didn't work. I rebooted it and then it was fine.

I'm glad he fixed it.

So I remember here in California, it was early morning on Saturday. I woke up, I turned on my T-Mobile G1 phone, I opened Google News and I saw the headline that Google flagging every search result as dangerous. And I thought, huh, that must be this other team that is doing—

It must be me.

No, I actually thought it must be— there's no way it's us. Because we had put a lot of guardrails in place that we said we should never put a pattern that would flag every .com domain, for example, but we never checked whether we would put a pattern that flags Every single domain in our list. And so that's exactly what had happened.

That's wild.

Then I remember by next week on TGIF, someone had asked the question. This is Google's internal all-hands, right?

And listeners, it doesn't happen on Fridays. It stands for Thursday Google Information Forum because it happens on Thursdays.

That's right. But back then it was a Friday afternoon, actually.

Oh, this is before we changed to be more global friendly on the timing.

That's so funny.

This really does go back.

It is back then. So someone asked, have we considered turning off safe browsing from search? Because I mean, this was a huge outage, right? This was basically Google was down. There weren't a lot of other products back then. I believe it was one of the founders that answered. We considered it, but we would have millions and millions of users with infected computers, and we don't allow that. That's not acceptable.

That's a huge vote of confidence for you.

It was huge. And you can imagine the morale on the team was terrible that whole week. We brought Google down. But since then, precision has been our number one priority. We're not probably the at the 3rd generation of systems that we have built, specifically not looking for bad things, but looking to prevent, to detect, and to revert any potential false positives that make it into the system. And it took probably in the order of thousands of SWE years of effort over the last 15 years to find the right data across the web that would allow us to estimate the risk of false positives. So it's a whole set of different classifiers that basically tell you, if you were to block this, are you going to have a bad time?

That's amazing. Thousands of years of SWE effort. Incredible.

Would you say that you sort of answered this question by saying, oh, we have the hard problem and we solve this problem by working even harder. But, well, I'm summarizing, but would you say that you had to sacrifice false negatives in the sense that you would see a website that's likely suspicious, but you can't quite call it, and then you say, this is okay, right? So you are probably sacrificing false negatives sometimes. So you would not block them. You would rather not block a benign site rather than, you know, miss to block the malicious site. Sorry, I'm rambling, but—

Yes, yes, that's definitely a trade-off. There's not a lot of magic when you pick a threshold on your ROC curve. Exactly. That's one of the trade-offs. And the other trade-off, as I mentioned, all of those sue years, right, that went into building these protection systems, there's a missed opportunity of putting those engineers on false negative reduction.

So it pains me to do this, but we need to move to our traditional closing questions. Aside from get a phishing-resistant FIDO key, what advice do you have for users to protect themselves against malicious websites? And 2, do you have recommended reading? I love people's recommended reading. It's always so interesting.

But you do need to get that key though, right? To answer before Panos answers, you do need to get the phishing-resistant FIDO key or a Titan key. Like, that's not— we are saying that.

We are. I would say there's 2 segments. There's your listeners and then there's probably everyone else. For your listeners, I would say, yes, get the FIDO phishing-resistant key. And if you can install an application allowlisting on your kids' PC, that's probably a good idea as well. To everyone else though, right, like your friends, your family, your high school friends on Facebook, I think I have a little bit of more actionable advice.

Oh, what? Please.

Ask them to go to their favorite search engine, whatever that may be, and type enhanced safe browsing. This will take them to a website that offers you an option to get our best protection ever. This leverages richer signals from the website you are on, and we are able to give you a more real-time verdict on the site you are on. The default Safe Browsing implementation is very privacy-preserving, but it does make some trade-offs—

Hmm.

—when it comes to how quickly can we update the list and whether or not we can find a new threat, classify a new URL that we've never seen before. Enhanced Safe Browsing gives you the option to bypass this, to make the right trade-offs. And if that's a choice that you like, it can get you up to 35% more protection for phishing.

Amazing. All right. And recommended reading?

So, I haven't been reading a lot of exciting things recently. I think the last book I'm on is called Foundryside, and it's a science fiction book, kind of fantasy.

You'd recommend it, right?

Would you recommend it?

I would recommend it. There's a little bit of hacking going on in a very different environment, though.

That's awesome.

Now I'm thinking, hey, maybe I should read it because it sounds like fun.

Now we have a book recommendation that's not the SRE book. So that's really nice.

Or not Dune or not whatever else is—

Panos, thank you so much for joining us today. Really appreciated getting to hear about the evolution of Safe Browsing.

Thanks, Tim and Anton. Have a great day.

And now we are at time. Thank you very much for listening and of course for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikin and Tim_Pico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.