EP131 A Deep Dive into Google's Assured OSS: How Google Secures the Software You Use

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts. as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button in your podcasting app of choice. You can follow the show and argue with us and the rest of our Cloud Security Podcast listeners on LinkedIn. Anton, we are talking about something that I think surprised both of us today, where we were just like blown away again and again by how cool it was.

Yes, and I think that there's only one episode when we tried to talk over each other with the same sentence.

Twice.

I'm confused. Yes, and the first time it was both of us trying to say, but wait, I'm confused.

And what was it the second time?

The second time I don't recall because the confusion really was interesting, but ultimately we got unconfused.

No, the second time was, wait, that's so exciting.

Yes, I think we both said, that is cool.

Yes, that sounds right. All right, well, listeners, with us just gushing about this topic, Let's turn things over to today's guests. I'm delighted to introduce today's guests. Today we have 2 guests joining us. It's a double feature for us. We first have Rahul Gupta, the product manager for Assured Open Source, and his engineering manager counterpart, Himanshu Khurana. Guys, thank you so much for joining us today. I want to start off, we're just on the heels now of our annual summer strategy session. And at this, one of the coolest One of the coolest stats I heard, and listeners, I got to hear stats for all of the products in Google Cloud Security. One of the coolest stats I heard was that of the packages that your product covers, you've discovered 50%, more or less, of the CVEs reported in those packages in the last year. That's simply astonishing to me. So how did you make this happen? That's amazing. Tell us about this, like, flurry of CVE discovery you're doing.

Yeah, Tim, this is correct. Security testing is an important part of the overall assurance we provide for Assured Open Source. When we started last year, we invested a lot in our security testing capabilities. There are multiple mechanisms we use, but one of the most common ones is a technique called fuzzing or fuzz testing. Fuzzing is a special kind of testing where you try to inject malicious inputs into the APIs of an open-source library. And then, you see whether you are able to crash that library or you are able to get some privileged access done. So just to give you a sense of the investment, initially, we covered, like, 7 out of CVE Top 25 type of vulnerabilities.

Uh-huh.

And then, today, we cover 17 out of Top 25 CVE type of vulnerabilities. This includes things like remote code execution and other kinds of code injection attacks. It's an intensive process where we first try to understand the API surface of a library. Then we write test cases, and then we continuously test these libraries using Google fuzzing infrastructure, which is one of the largest infrastructures in the world for fuzz testing.

Well, it must be if you're able to find half the issues out there for them. That's, that's really cool.

Yeah. So this happened during last year. We were able to find around 48% of the CVEs reported on the packages we curated. In fact, one of the vulnerabilities we found had a CVSS score of 9.8.

That's a lot.

And this was a remote code execution attack possible in one of the popular databases called Hypersql. And, you know, it's also one of the top 200 popular softwares on Maven Central. Yeah. And this was fixed in collaboration with the maintainers, where maintainers agreed to this, and they fixed it. within the certain disclosure window. Yeah. So very exciting.

So now that gives me an interesting thought is I hadn't thought about the maintainer side of this. Is it now the case that Google's just like showing up to a bunch of maintainers being like, hey buddy, you got this like long list of issues? What's the relationship management look like?

Yeah. So we have a pretty strong relationship with many of the maintainers. So it goes both ways. For example, when cases where maintainers agree and they want to fix it, they would do that and we help them with the The details of the vulnerability we found, where we found it, and what's a possible fix. And in some cases, maintainers say that, hey, I mean, it's a lot of work for us. We don't have bandwidth for that. So we actually go ahead and do fix the vulnerabilities, raise a pull request, and then they accept it.

All right. So listeners, I did something kind of mean. I started talking about the real deep details of something, but I thought it was so interesting. Anton, do you want to like bring us back to square one maybe?

Yes, indeed. And I think that it's the topic of the conversation today is, of course, assured open source. So, I wanted to, while on this podcast, we kind of hesitate from asking, yeah, hey, product manager, define your product, because sometimes we get a 6-hour talk as a result. But in this case, I think it's very useful. So, how do you describe assured open source? And our side question to this is, well, it's a commercial product, we charge money for it. So, what gets better if customers use it? I think I already know what it is, but I'm curious.

Yeah. Assured OSS is a managed service that customers use to secure the ingestion of open-source software in their ecosystem. I think the fundamental idea behind Assured Open-Source Software is how can we help customers make Google's software supply chain expertise their own? Can customers use the same set of packages that Google curates, maintains, continuously tests and secures, and uses for itself? Right? How can we help customers demonstrate the integrity of software ingredients to different regulatory and compliance authorities? And within Google, we secure open source through a combination of people, culture, policy, and processes. And at Google, we've spent a lot of time building and optimizing this system. And as a result of that, we think that we now have a system that helps us manage the open-source software with an understandable amount of risk. So Assured OSS is essentially externalizing this internal system, which will help customers obtain their open-source packages from a trusted and known supplier where all the packages are built from a mirrored source. They'll know more about the ingredients with the help of Assured SBOMs that we provide in industry-standard formats. They'll be able to demonstrate confidence in the integrity of their ingredients through signed tamper evidence provenance and reduce the risk in their open source usage because Google is actively scanning, finding, and fixing new vulnerabilities in these curated packages. So in essence, Assured OSS gives customers the confidence that the open source software that they're using has not been tampered with And can be securely traced back to source, something that is difficult, if not impossible, to do with most software today.

But it's not just traceability and SBOMs. We also fix issues. We find and fix issues, as you described, right? So objectively, our package for the same software is more secure than the classic open source, right? Is this true?

Yeah, we are not doing any fork management, right? As Himanshu mentioned earlier, we are upstreaming all the vulnerabilities and the fixes for those vulnerabilities by working with the maintainers. Again, this has been a request from a lot of our customers. Can we give them vulnerability-free packages? And it's an area of exploration for us. But for now, we want to continue to leverage our upstream community relations to ensure that the broader ecosystem benefits from the work that is being done as part of Assured OSS.

So this is what confused me.

I apologize for my confusion.

Oh, no, this is what confused me too. And listeners, Anton and I just talked over each other, both being confused. But I want to talk about being confused today because Anton's always confused. I get to be confused sometimes. This sounds like you're building lighthouses or street lamps. You're creating public good that anybody can benefit from, which is, you know, 2 thumbs up, great job, pat on the head. But how do you make money at this if you're upstreaming all the fixes? Like, what are your users actually paying for now?

Yeah, so great question. And you're right. But just to set the record straight, I think firstly, Assured OSS is currently free. And we have a curated—

Oh, great.

1,000 most commonly used Java and Python packages that customers can use today. Now, regarding Assured OSS discovering and fixing vulnerabilities with upstream community, as I mentioned, it benefits the broader ecosystem, not just because we find them, because we actually fix them. And our goal with vulnerabilities is to be first to find and first to fix. But securing your software supply chain goes way beyond mitigating vulnerabilities.

Mm-hmm.

The attack surface is huge. And this could range from compromised repos, from where people are sourcing their open source packages from, compromised dependencies of those packages, malicious or typosquatted packages, and compromise of the process through which the packages were either built or distributed. And Assured OSS helps with all of that. On top of it, there's another aspect of compliance. We're seeing more and more regulatory scrutiny emerging in Securing software supply chain, the White House's executive order, NIS regulations in UK, Cyber Resilience Act in Europe, etc., all require customers to demonstrate more and more understanding of their software ingredients. And Assured OSS helps customers meet these compliance and regulatory requirements through enriched metadata, which we provide with Assured SBOMs as well as provenance.

Okay, so Anton, I know we've had a swear jar for a long time for mentioning your blog, I think on these supply chain episodes, we might have to create a swear jar for mentioning the EO. I think everybody's touching on it these days.

I mean, it's an influential piece of documentation or guidance. So, I can see that. But it's also, I think, your answer. I'm not confused anymore in that sense. I think that your question was exactly what I wanted to ask is that if we are not, I was afraid we are forking the packages, and I'm happy that we are not forking the packages. But so, then people ultimately are paying, well, will be paying money to have a trusted source for software and to reduce the software supply chain risks specifically, rather than to have less vulnerable software from us. They're not paying to have less vulnerable open source. They're paying for— well, they're getting the less vulnerable open source from everybody, but the source is trusted.

Yeah, essentially, you're right. Like, the packages are gonna be more secure because the attack surface gets reduced with all the curation and maintenance and the building of the packages that we are doing. Right. The particular aspect of vulnerabilities is where we feel we should continue to benefit the broader ecosystem because that's just one aspect of securing your software supply chain.

Okay.

And I really like that. I really like that there's the SBOMs and the secure build and the typosquatting prevention. There's like all kinds of good stuff that's happening here. So this, at the same time as you're creating all this good for the world, I'm actually jealous. Like the work I do, I don't get to have these lighthouse benefits for everybody. I only benefit the people that pay for my lighthouses. You're, like, way cooler than me. This is great.

Actually, in this case, I think I understand. I wanted to ask a question, how can organizations start using assured open source? But in essence, there's no specific thing. Like, if you use an open source package on the list, then you already use it, right? Like, so, can you talk a little bit about how do I start benefiting from an open source if I'm an organization?

So yes, you might be using open source packages already from public repositories, but then there is a different build done by the Shared Open Source team. And these packages are hosted separately in another repo, which you get access to.

Uh-huh.

The serving interface is exactly the same as any public repository. So all you need to do is to point to a different repository and you'll get a different package built by Google. It will have a different hash, different signature. It would use the same source, but the whole other things about the supply chain, like Rahul mentioned, would be secure.

And you're reporting Salsa levels around all that as well?

Yeah. The builds are currently Salsa level 2, and we are working on Salsa level 3.

Nice. So, you're going from medium to hot. That's great.

That is actually quite impressive. And I think that I get that the trusted build argument has been quite broken in some recent incidents in some places. So, I think that somewhere around the discussion of Azure Open Source, I've heard you mention that we guarantee the security. And I think in the industry, I think in our industry, a lot of people are kind of nervous when others say, oh, we guarantee something. So, can you explain that? Like, what do we— do we guarantee anything? And if yes, what do we guarantee?

Yeah, I think we guarantee the integrity of the artifact, right? And we provide all the attestations around it. So, just to look at a software supply chain for a dependency or an open source package. You have the source code, which we actually mirror and run additional security checks on, such as origin validation. And then we take accountability to build that package from source, and we provide signed provenance for it. This provenance can be validated by customers, integrated as part of their compliance mechanisms. And finally, we also take ownership to securely distribute those packages to avoid meddler-in-the-middle type of attacks. So that's the assurance that Assured OSS provides, that you will have complete transparency for any open-source dependency that you're using in your software, and you will be able to verify with tamper-evident provenance.

Oh, that's really cool.

That's really cool, actually.

Yeah, we're both like equally impressed. We're first confused and then impressed. This is a good episode. We're learning stuff today. So one of the things I love about working at Google is like you get to go around and find this weird alien technology sticking up out of the ground and turn it into a product. Where's the Google magic in all this? Is it the fuzzing that's Google magic? Is it the signing systems that's magic? Like, what have we rolled out to the world that was previously like buried in the desert under the sands of time?

Yeah, I mean, at a high level, there are 2 things, Tim. One, we follow our own best practices we created for internal open source usage.

Uh-huh.

Which is like internally immutable mirrors within Google to prevent any tampering of commits. And we test the source code, we build with Salsa compliance, we generate SBOMs, we sign the artifacts using cryptographic keys, we make the signatures available for verification of binaries, and then finally, we host these packages in a non-public repo for customers to download. And the second magic is our ability to do these things at scale at Google. That's what I would say.

So, the scale is some Google magic. I'm trying to imagine what the fuzzing looks like for all these libraries. How are you scaling that out?

Yeah. So, fuzzing is currently a manual process. We write test cases, like I said.

It's manual?

Writing the test cases.

Oh, manual, not in the sense of fuzzing itself. I think planning is manual, but fuzzing, of course, is automated, right?

Yeah.

It would be a little crazy to fuzz manually.

So, fuzzing has 2 parts. One is A one-time work about writing test cases, which is a very intense process. You identify the API surface, write those test cases, and you're done. And then there is a part where you run these test cases automatically every day, all the commits, and then see what bugs it is finding and then triage them and then work with the maintainers to get them fixed, report CVEs. We are also working on trying to scale this whole process through automation. But it's a very intense process right now.

Yes.

I bet it is. Are you also doing cool things like taking where you find the bugs and the kinds of code paths that it's in, or the method signatures or something, and training AI on where bugs tend to show up and then focusing your efforts with the AI? Is there some AI magic we can do here?

We don't have a Square JAF AI yet, right? We need to make one, right?

Honestly, I should put my whole wallet into that for that sentence. But I really think there's something cool there. Like, are you doing cool AI stuff yet?

So there is a, there is a very cool part we do right now where when we find a bug, we try to pin it down to a commit which caused it. So for example, think of a binary search. Let's say you found a bug in version 7 and you don't know when it was introduced. So now you check version 1 and then 3 and then 5 and then 4 and oh, version 4 caused this bug, right?

Mm-hmm.

Now you know what changes went into the code. which caused this bug, and then it is easier for the maintainer to actually go in and fix it.

I can see how you can train ML on this later on, right? Like, you can train ML on this later on and some magic will happen, probably.

So that's the second aspect. There is an ML part, so we are also working on it where we are trying to train models to, first of all, look at an API and decide where we should write first test cases for maximum coverage.

Ah.

And then there is another initiative we have where we are trying to find vulnerabilities using ML. These are all work in progress, not launched yet, but yes, this is what we are doing. Yes.

And now that to me is really interesting on the threat side that you're actually getting back to the commit that put in the issue. Are you seeing cases where it was deliberately introduced? And are you seeing the same actor do that to more than one project?

Not exactly in those packages which we tested so far. But yes, I think there has been incidents where people introduce bugs deliberately. For example, I remember an incident where in an npm package, a developer introduced an infinite for loop. I think it was Colors and Faker sometime early this year. So those things also happen, but we haven't discovered anything like that so far.

Well, I look forward to your first, like, nation-state-introduced bug that you discover. I can't wait for that.

So, I'm actually kind of more curious about the ML because frankly, that some of the stuff, we never know what the algorithm would pick from if you have enough of these fed into the system. Some, I mean, really even more impressive magic may happen if you train the system from, say, connecting to other things. You might discover things that humans would never discover. And to me, that's actually quite exciting.

Yeah. Yeah.

So to turn this to badness, it's a security podcast. So in recent weeks, months, a lot of people were kind of talking about supply chain attacks. And sometimes I hear that people sort of like, I don't know, supply chain wash other attacks. Like there's an incident and they say, oh yeah, supply chain was involved. But in reality, it was sort of more of a traditional attack. So how do you think about like a real supply chain incident? And what are you seeing, maybe hearing from clients? about the supply chain threats?

Yeah, so we can talk about some examples here. First of all, let's look at the supply chain for an open-source dependency. There is a source code, then it gets built, then you have some dependencies which get used or packaged, then you have hosting, and then finally it gets to some production machine during the deployment process. So anything can go wrong during all these steps. And these are called supply chain attacks. So first of all, like we said, like the source code. Now, there could be vulnerabilities in the source code, intentional or unintentional. That's one. Then, the second part is build. Now, SolarWinds attack is a classic example of a supply chain attack where an attacker hijacked the build process and then placed a backdoor inside a legitimate digitally signed software.

Okay.

And those updates were pushed to the thousands of customers over several months. Now, imagine that an open-source library is being built by an individual developer on their laptop.

Uh-huh.

What if the laptop is compromised? Right? And then dependencies, right? I just spoke about the npm example where there was a malicious code injection to create an infinite loop in a couple of libraries. I think it was Colors and Fakers.

Uh-huh.

It broke thousands of applications across the world immediately. So, those are examples of compromised dependencies. Then you have packaging and hosting, right? So, PyPI disclosed last year that there was a phishing attack which could have caused hundreds of malicious packages uploaded by somebody.

Hmm.

And just a month back, PyPI temporarily paused new users and new projects because they were seeing a very high volume of malware. So, there are people who are trying to upload malicious packages. on PyPI directly.

Okay. So there really is like true supply chain badness out there. That's good.

Exactly.

And it's also not all point. Traditional security controls that people think of would be quite useless against this because— and that's— I'm not— sometimes vendors say it like it's FUD, but in this case, it's just absolute truth that if you do things normally and you do things well, You would miss it. Yeah.

Rahul, what are you thinking?

Yeah.

So just to reiterate the importance of supply chain security and the attacks that have happened, open source security and software supply chain attacks have grown exponentially over the last few years. Mandiant's mTrends report highlighted that 17% of all breaches last year started with a software supply chain attack, and it became the number 2 initial infection vector behind exploits. And the nature of these attacks has also fundamentally changed as injection of malicious code through various attack vectors is a different and distinct threat from, let's say, the exploitation of vulnerabilities, right? And now when you look at and compare and contrast how typical applications are built and shipped, it's through a combination of closed source, which is the code you write in-house, and open source, which is what developers generally get from public repositories.

Mm-hmm.

And industry research shows that nearly all applications include an open-source software component, and almost 85% of open-source components have one vulnerability, with the majority of them being found in transitive dependencies. So, just wanted to put that out there so that people understand why supply chain security is kind of emerging as a top-of-the-mind concern. for our customers and the industry in general.

Nope, makes a ton of sense. So on this cheery note, listeners, we are at time. And Himanshu and Rahul, I have to ask you our traditional closing questions. First, do you have one tip to help people improve their open source security posture? And 2, do you have recommended reading? And the recommended reading could be anything, not just the MTrends report, by the way.

Yeah, so I think Like we talk to a lot of customers on a daily basis, and the biggest challenge that they have is they don't know what they use and where it is used in their system, right? So knowing your open source usage is, I think, a critical baseline that everyone needs to establish in order to kind of detect and fix issues to reduce the supply chain risk.

That's your classic, you can't manage what you can't measure. You can't secure what you can't see.

Exactly. And in terms of the reading recommendation, the Salsa blog that we have at slsa.dev is a great resource for learning about what's happening in the world of software supply chain, both in terms of attacks, practices, and what people can do in terms of safeguarding themselves against it. So I would definitely recommend reading that.

I love that. Himanshu, what do you got?

Yeah, and my tip would be from a You know, software developer perspective, when I would make a decision that I want to include a specific open-source software into my code, I should check how is the health of that particular software package. There are many things available like OpenSSL Scorecard. Is it being well maintained? You know, do they fix vulnerabilities quickly, et cetera, et cetera? Because if I make a wrong decision today, it's gonna cause a lot of tech debt.

Mm-hmm.

for me in future, because then I'll have to find replacement, patch, et cetera, et cetera. So know your package before you start using it. And then, on the recommended reading, in fact, I listen to 2 podcasts. One is Google Cloud Security Podcast, big fan.

Aw, shucks.

And then, there's another one called Open Source Security Podcast by Josh and Kurt. So again, if you listen to those 2 podcasts, you will know a lot about what's happening across the industry. You know, tons of knowledge and cool tips.

Yes.

Oh, Himanshu, Rahul, thank you so much for joining us today. Anton and I both, now that we understand it, clearly love what you're building, and we are delighted to see what happens next with your product area.

Thank you so much for having us here.

Yep.

Thanks. And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikin and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Secure ready.

Secure ready.