Linking Up The Pieces: Software Supply Chain Security at Google and Beyond

Security. Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Tim Peacock, the product manager for threat detection here in Google Cloud, and Anton Chuvakin, a reformed analyst and member of the cloud security team here at Google. You can find and subscribe to this podcast wherever podcasts are sold and distributed, as well as at our website. cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show and argue with your hosts on Twitter as well, twitter.com/cloudsecpodcast. Anton, we've got a fun episode today. We are talking about supply chain security, aren't we?

Yes, we are. And I want to start in somewhat unusual manner with some FUD.

Some FUD?

Yes, exactly. So when I first read up about software supply chain issues, I kind of thought, that is really scary. Yeah. You can hack somebody who provides software for you, but if you can't hack them, then you can hack somebody who provides software to somebody who provides software to you. But if they're still secure, you can hack somebody who provides software to somebody who provides software to somebody who provides software to you. In any way, you're still hacked. You're still done. Sorry if I'm using the word hacked to mean criminally compromised, but the point is that it's just scary. I feel very powerless when I think about it. Like, you can't do anything. It's not you. It's the company that supplies the company that supplies the company that would be the cause. So it's really scary. I'm like genuinely terrified. So what can we do about it?

That's one of the things that we talk about in this episode is what can you do about it? And part of it, I think, is one of those cases of, you know, you can measure what you can manage. And so we're going to talk about some ways of measuring this and some, some ways of thinking about the problem systematically. We'll talk about different levels of Salsa. And as a hint, they're not, you know, mild, medium, and hot. But we'll also touch on briefly a topic called SBOM. Which I think a lot of people have been talking about lately. And there's a lot of good stuff in there.

Yeah, but it still remains a terrifying problem. You're giving like this cold-blooded, even-minded view, but I'm giving the terrified view. Instead of thinking, can my software be compromised? You have to think many levels before you even got access to your software, right? So, and it's not only for software companies. I use software, but like who made the library that some provider to that software supplier uses? And this is mind-boggling.

For sure.

I'm sorry, I'm still terrified.

No, it's a terrifying problem and it's not exactly a new one. I mean, we've had the problem of trusting our supply chain for a long time. Like, untrusting trust is kind of the root of all this. Yep.

1984, right?

Yeah. I mean, 6 years before I came into this world, we had this problem.

Exactly that. But here's the funny one. Maybe it's like the Google-sized problem to solve, right? Like, it's so terrifying. It's so scary. Maybe Google is the right place to solve it.

I think that's an excellent transition into our guests. So with that, I'm delighted to welcome our guests today. We're joined by Aparna Sinha, Director of Product Management here at Google Cloud, as well as Eric Brewer, a Vice President of Infrastructure and Google Fellow here at Google. Eric, my first question is for you. It's perhaps what you might call a softball, but for people who've been living under a rock or those who don't think much about software security, what is software supply chain security and how is it different from other kinds of supply chain security?

Well, the classic supply chain came about from the idea that when you buy some piece of equipment, it has many components, And you might have a list of, or a bill of materials, of the things that go into that device. And likewise, modern software is built with many components, and so there's software inside your software, and we'd really like to know where do those pieces come from and should we trust them.

Got it. So it's really, at the end of the day, about trust.

Absolutely. It's great that we can take software so many places— open source, libraries, packages— and that makes us More productive, improves quality of software in general, but it means we're trusting a lot of different sources. And there's lots of different ways to attack that supply chain. And it's become a bigger issue in the past year. And so now we have to kind of take it much more seriously.

So in light of this, you mentioned attacks against software supply chain. And so what kind of organizations need to care about this? It sounds like a large enterprise concern or something like a sensitive agency would care about. Who do you think should care? Like, what kind of organizations must have it as top priorities? For everybody? For large enterprises? And maybe give me a few examples that you've heard from users.

Well, all organizations that use or create software should care about the security of that software. And these days, software is everywhere, especially post-pandemic, because most companies have moved to digital channels and accelerated their digital transformation. So as you know, every company is becoming a software company. Specific things that we're hearing from customers these days, users are quite concerned about security risks, especially as they have distributed and remote work teams and they're moving to online channels. We're especially seeing in the public sector a very strong stance about making secure software supply chain a matter of national security. We're also seeing state and local governments moving very quickly and looking for standards and guidelines to use as security blueprints. Overall, we're seeing security budgets grow in financial services as well as large enterprises everywhere, and just strong interest in automation and DevOps and managed services to lock down the security concerns.

So Aparna, you mentioned, you know, national security, and I know there was an executive order recently that mentions both supply chains, and Eric alluded to bills of material or software bills of material. I guess this is a toss-up question for you two. What's the relationship between what we're doing here at Google and what SBOM is, and how does it all fit together?

Well, it's a tricky question because the software bill of materials was originally designed essentially for packaged software. So, you buy some software off the shelf, and you might wanna know what pieces go in it so that if one of those pieces has a vulnerability, you might be able to do something about it. But that's a very '90s view of software, and Google's own internal workings and cloud in general, of course, is much more dynamic, and we're building software and rebuilding it all the time and doing incremental upgrades and rollouts. And so, we don't really have a bill of materials in the same sense, or at least not the same need of putting out a packaged piece of software. But the concerns are valid, meaning we'd like to know exactly how every binary was built, what went inside it, why do we trust it, what exact source code was used, what tools were used. In fact, the NTIA just released their requirements for SBOMs, and one of the things it says basically is, Clouds don't really need to follow exactly this because it's not perfectly appropriate, but clouds should in general try to meet all of these goals, meaning knowing where the software comes from and whether it has vulnerabilities or not. And in fact, we already do that internally. And now the goal is to make that easy for our customers to do as well.

That makes sense. But to kind of bring it more to life, you mentioned threats and you mentioned certain actors, certain companies. So can somebody lead the audience and of course us as well through kind of a quick threat assessment of a supply chain issue? For example, For example, does it mean that it's like a criminal hacker may somehow disrupt the supply chain, maybe through open-source software, or does it require a government agency from a perhaps unfriendly country? So just give us a bit of a threat assessment about how a supply chain issue may arise and what are the actors and what are the realistic threats here.

The one that got the most attention originally was SolarWinds, but there was another one right after that called CodeCov, which was an open-source attack involving GitHub, among others. And that one's more interesting. I'll get to that in a minute. But there's also some of the recent ransomware attacks have also used supply chain attacks as part of their attacks. So it's interesting to see that the criminal actors are now copying some of the techniques originally used by nation states, Russia in particular. And so this is a pretty big threat model for anyone that's at risk of ransomware, in addition to anyone that a nation state might like to attack. But what makes supply chain attacks particularly nasty is they're really attacks that lead to attacks. And what I mean by that is the CodeCov attack, for example, was essentially inserting a bad version of Bash, which is a scripting language used by a third-party tool of GitHub. So, not GitHub's fault directly, but CodeCov is a tool that lets you look at code coverage on your code, and that tool uses Bash, and the attackers made a bad version of that that enabled is to steal credentials. But that steals credentials from everyone that uses CodeCov, right, which is many customers. And once those credentials are stolen, they can be used for many other purposes. So you can think of the original attack as a credential factory that produces credentials, and then those credentials can be used for further attacks down the road. And that's what most groups in the end get attacked is because their credentials were stolen in a supply chain attack, and then their primary software or products are now at risk.

These are seemingly quite different attacks, and they all involve some degree of amplification through distribution or an upstream compromise. How do you know a supply chain attack when you see it, and how do you not end up with everything becoming a supply chain attack? I'm struggling with a crisp definition here.

I kind of think it has this 2-stage aspect that's pretty fundamental. The first stage is something in the supply chain, typically to either insert an attack point Or to steal a credential. And I think of that as kind of as a malware factory because it produces continuously, essentially, as long as it's operating. And then afterwards, you can use those attacks, those credentials you got, to do second-stage attacks. And that's exactly what happened with both SolarWinds and CodeCov, where the first attack is attacking SolarWinds, and then SolarWinds, because it's used inside enterprise networks, was able to do further attacks from there.

Got it. So maybe to make things more concrete, Aparna, how is Google solving these problems internally? And do you think we're succeeding at solving these problems internally?

Well, Google has had a very strong approach towards software supply chain security. And in fact, we've had mechanisms in place for the last 8 years or more that secure our own software supply chain. And there are several practices, including code review, extensive code review, generation and storage and checking of provenance, which is what's the origin and what's the chain of trust of all software, as well as policy enforcement, are 3 big pillars of our supply chain security capability that we use internally. And of course, all of that is backed up by a secure infrastructure, secure networking, and secure operations, et cetera, as well. But the software supply chain internally that our developers use is secured through these 3 mechanisms primarily. And I do think that it has been successful, proving to be a source of some of these external standards discussions that Eric is leading.

So, in addition to the standards, has it turned into, say, tooling in Google Cloud or other things that we've released to the community? Do we have a great user story of how somebody switched over to this and foiled the adversaries?

Very much so. Yeah, I would say that there are some pillars of our product capability that have emerged from our internal experience. You know, our binary authorization capability, our build system, and of course, the way that we build all of our managed services is heavily utilizing the experience and the internal capabilities that we have built. We talk about the SALSA framework, that's an emerging standard. You know, our cloud build system provides the kinds of provenance that are required for SALSA Level 1. Our binary authorization service is very much parallel to the kinds of policy enforcement capabilities that we use internally, which is that we generate attestations at various points in the supply chain, and then we check them and either enforce or alert based on those policies using binary authorization. In terms of customers that have benefited, I think, yes, many, many customers have benefited. With security, it's the avoidance of problems. And so there isn't a big story about the avoidance of problems that I would share. But there have been several security vulnerabilities that Google has simply protected its customers from.

To play a bit of a devil's advocate on the same topic, we can certainly tool up and build tools and build frameworks, but our customers who may use open source tools, Are they expected to approach open source as inherently suspicious then? Like, how would you not become too paranoid about all the libraries, all the software you use? Like, sorry for taking a bit of a curveball direction, but like, what's a normal company to do? What's a small bank to do?

I think, you know, open source software is the future of modern software development, and rapid iterations, continuous deployment is the future of modern software deployment. We have seen through a research study that we conduct called DORA, DevOps Research Association, over the last 6 years that highest-performing organizations are continuously upgrading their software at a very rapid pace. And that actually, of course, means that they're able to achieve better business results because of software development agility. But it actually also means that they're able to achieve a higher level of security. And that's kind of counterintuitive. So, yes, there are many, many open-source dependencies. You're relying on code that isn't under your control. But that is why there are many tools being worked on, both in the open as well through some of these foundations. The Open Source Scorecard is an initiative that we are involved in. And of course, Open Source Insights, which is a website that we recently introduced that gives you a transitive dependency graph of open source packages and has a database of vulnerabilities, as well as licensing and other security-related data, and provides the ability to, you know, do version comparisons so that you can be more careful about checking your dependencies and which packages you use. And that, in conjunction with the Open Source Scorecards, Gives you a better mechanism for deciding. But there are a whole host of other things that we're doing. For example, in the Go language with fuzzing, as well as shifting left on the scanning process, we're providing developers the ability to essentially find these vulnerabilities, identify which ones are important, and mitigate the risk earlier in the development process.

So it sounds like Google made it easier even for normal people, for non-Google-type companies. to get some of the benefits of software supply chain security without doing all of the work we've done. So, we have frameworks, we build tools, we build visualizations for them to track stuff. So, basically, they can be directionally almost as secure as we are despite not spending all this money and time, right?

That is certainly our effort. That is certainly what we're trying to do. We have certain learnings, and it's clear that many other companies can benefit from them. We have introduced certain products and services. I would say that it's a process and it's a journey, and we have much more work ahead of us.

I would just add that in the long term, because it's a very complicated problem, most companies probably should not try to solve it directly themselves, meaning that if you're doing your own builds on your laptop using whatever repositories that are on the internet to get your source code, that's just not gonna be a secure process. And so, what that really ends up meaning is you'll have to have trusted repositories of source code, probably managed by someone else, Google will certainly be a candidate for that. You'll want to have a build process that is secure and trusted. That's what we do internally, especially for production workloads. And you want to have some way to get notified of vulnerabilities that are discovered. So, there's a bunch of pieces here that are coming together. But I think my main message is that most companies should not directly try to solve these things themselves. It's actually quite difficult to do well, even for a big player like Google, and therefore working together with our customers is going to, in the end, be the right solution.

I've definitely seen cases where this is hard even for us to get right. Perhaps another softball question, maybe it's just, you know, striking me because it's mid-afternoon and I'm hungry, but you mentioned SALSA. What is SALSA?

SALSA is a proposed framework for how to deliver invariants about how things were built and why you should trust the inputs and the code and the build process. It's based on what we've done internally for our own supply chain, But it's a bit abstracted and more amenable to open source because, you know, internally, we have a monorepo, we have a lot of control over the systems and tools that people can use. So, it's not as flexible as open source. So, Salsa is our interpretation of how to take those ideas, apply them to open source, and then to add to that kind of 3 tiers. They're kind of incremental in what promises they're able to make, where the Salsa 1 is just basic things about, Do you know what the build process is and what happened, up to higher levels where you actually know that code is reviewed and that the exact source code was known and kept, and various other properties that you'd like to have. So, it's really all about the invariants you want to be true about your overall artifact production.

I got it. And so, you said that it has levels 1, 2, and 3, not mild, medium, and hot.

It does not, although maybe that was a mistake.

That feels like a miss to me. So, this whole episode came about because we have an upcoming S3C Day. What is this event and what should listeners know about it?

This is a great event. Eric and several of the folks on the Google security team have been working on standards that we're developing with open source foundations, as well as working with, you know, the government agencies. We wanted to have an event that features an awesome panel chat with Eric Brewer, Phil Venables, our Chief Security Officer, as well as one of our amazing customers, which is Shopify, to talk about what some of these emerging trends are and the thought leadership around this area. So, that's the first part of the security event. In the second part, we'll be talking about some of these exact capabilities that Google Cloud provides and how we can help our customers achieve greater supply chain security. And that'll be through some product announcements as well as through a number of cool product demos. So, for example, this Open Source Insights website that we talked about that shows you these dependencies, you can see that graph and we'll give you a full demo of that. that, this capability around Salsa with Cloud Build, we will be demonstrating that as well, and binary authorization and the policy frameworks. So those are each deep dive tracks at the event. And we really look forward to having some live Q&A with our customers as well.

Perfect. Perfect. So it sounds like an event not just for people who already know how to secure software supply chain, but it's kind of useful for people who are still figuring out the threats, figuring out the defenses, figuring out the approaches. So it's kind of education and not only for the enlightened, right?

Absolutely. This area is very new and emerging. The securing of the software supply chain is an unsolved problem and it is a work in progress. So, this event is very much introducing the challenges, introducing the frameworks for thinking about this, and the initial kind of guidance and solutions that are out there.

Perfect. Thank you very much for this. So, as we are kind of coming to the closing time, we have our traditional closing questions to ask you guys. So, what is one practical thing for a company, for our audiences in this area? So, if I am maybe not new to security, maybe not new to cloud, but I'm very new to software supply chain security, what is one thing I can do? Is it adopt a tool, apply a framework, you know? And the second question is, any recommended reading? I mean, of course, we'll put the links to TELSA Framework and a few other things, but any other suggested reading for the audience?

We generally recommend the 4 steps towards understanding secure software supply chain. One is awareness. And so, we would really recommend following following some of the standards work, for example, by coming to the event and reading some of the articles and so on that we will share. Second is actioning, which is really using some of the Google best practices. And there are many ways to do actioning. I think one of the most practical ways to do that is to use fully managed cloud services because fully managed cloud services already kind of bake in the best practices. And if you use them, you don't have to do much setup yourself. The third is investing in automation, and the fourth is really investing in auditability. And that's, of course, for software supply chain, what you want to know what the chain of trust is, and you want to be able to audit that and you want to be able to enforce policy. So if you're just getting started, I think the first thing is to gain awareness of the standards work. The second thing is to start using some of these managed services and understand the best practices that are built into them. And then the third thing is to start to invest in auditability and automation.

Any suggested reading for the audience?

Yes, we have actually a Forbes article that is coming up by Eric and myself provide a layperson's view of this whole space. So, we will share that with you. And then, of course, we have some blogs that are coming up along with the event. So, we'll share those with you. We also have a Google Open Source Security blog that has had several very good articles, for example, on SALSA and binary authorization for Borg, that captures the history of how Google has attacked this problem in the past. Those are great background reading.

Perfect. Perfect. We'll absolutely put the links to these on the podcast site. As a quick reminder to the audience, when is the event?

The event is going to be live on July 29th.

And will it be recorded and available forever afterwards as well?

Yes.

Awesome. So listeners who are joining us after July 29th, 2021, you can find the event online as well. Eric, Aparna, thank you so much for joining us today. Listeners, thank you for joining us.

Thank you. Thank you very much.

And now we are at time. Thank you very much for listening and, of course, subscribing to the podcast. Also, see you all at the Container Security event. You can learn about the problems and solutions in software supply chain. We just mentioned it's a scary and terrifying problem. So, you can listen for this live on July 29th. And, of course, afterwards, it will be recorded on YouTube. Find links at the podcast URL. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, Or wherever you get your podcasts. Also, you can find us at our website cloud.withgoogle.com/cloudsecurity/podcast. Probably pretty easy to remember at this point. Please subscribe so that you don't miss episodes. You can follow us on Twitter twitter.com/cloudsecpodcasts, and of course your hosts are also on Twitter @anton_chuvakin and @TimPeacock. Tweet at us, email us, argue with us, and if you like or hate what you hear, we can invite you to the next episode. See you on the next Cloud Security Podcast, and of course, at our Container Security event on July 29th.

Security.