EP116 SBOMs: A Step Towards a More Secure Software Supply Chain

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts. as well as at our website, cloud.google.com/podcasts. This episode, listeners, extra special today, also available on our YouTube channel. You can watch us with video, and it's actually a pretty clean take, so you won't be missing out too badly if you only listen to the audio. If you like our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show and argue with your hosts on Twitter as well. twitter.com/cloudsecpodcast. Anton, this is an SBOM episode. And I have to say, this is the first episode we've done where when I told somebody I'd done it, they were disappointed they weren't the guest.

Really? But it was also, I think this episode is notable for something else as well. This is the episode where upon learning that we should probably consider doing it, my reaction was, what? No, this is a, I don't, did I say snooze fest? I didn't say snooze fest. I said something else, but it was kind of fairly unpleasant.

You certainly didn't give it the reception it ended up deserving once we recorded it, that's for sure.

Yes, absolutely. I was initially somewhat skeptical, bordering on pessimistic, bordering on mildly depressed about it.

But this turned out to be a super fun episode with some awesome tortured analogies. We get into like fundamental discussions of what's the role in the state when it comes to markets and can markets exist without a state. But listeners, we also talk about security and good stuff in the context of cloud security. So with those show notes, let's turn things over to today's guest. Isaac, thank you so much for joining us. It's my pleasure to introduce you. Isaac Hepworth, joining us live. You're a PM here at Google focused on software supply chain security. I want to get started by asking what kind of sounds like a softball question, but an important getting started question. Everybody's talking about SBOMs. How come, and what do they matter to today's company?

What's an SBOM? Wait, you wanted to ask what's an SBOM?

Yeah, what is an SBOM anyway?

There's a great kind of zeroth-order approximation you can use for SBOMs, SBOMs, which is, think of them as, I conceptualize them as the ingredients label on packaged foods. And hey, if I have a nut allergy, or if I have adverse reactions to certain ingredients in foods, I'm going to want to know which foods contain those ingredients. And so, SBOMs, as a capability for software, it's a listing of what are the components in this software. And you can think of it in the same way as an ingredients label, that, hey, if I have an adverse reaction to this particular version of Log4j, or this particular component of software, I can at least know about it when I'm consuming software with an SBOM. And I think the good analogy is that if I have a nut allergy, then ingredients labels on food aren't enough in themselves for me to stay safe, but good luck staying safe without them, right? There is the need to read the ingredients labels and respond accordingly and so on. But the ingredients labels is a metadata substrate through the supply chain, which provides useful information to consumers of software and helps them stay safe.

But so why does it matter to security leaders? I mean, I think I can guess at the answer, but like, why don't you guys give us an answer? So these are useful, but why are they useful for security leaders at an organization specifically?

So I think, I mean, for vulnerability management particularly, I mean, you know, so we're all familiar with, you know, late 2021, there was the weakness in Log4j. And I think, you know, when Log4j deployed with great breadth throughout the industry, and the first question that organizations wanted to ask themselves that were, Oh my goodness, it feels like there's some vulnerability or a weakness in Log4j. Where in my organization do I depend on Log4j? Which pieces of commercial software am I running that depends on Log4j? Which pieces of first-party software depend? What am I shipping to my customers that depends on Log4j? And SBOMs really provide the answer to that. I mean, in the world where SBOMs are ubiquitous and every piece of software has an SBOM, I can very quickly, as a vulnerability management practice, establish where is this weak component, or where is this risky component, or how should I think about risk in my organization? It gives you a picture of where particular components show up in your dependencies. And when you know those components are vulnerable to something, you can then make reactive moves to protect your organization or your customers.

Okay, that actually does make sense. And I think that your food analogy example was very telling, but I'm hearing that some software vendors perhaps don't really like the concept because just like with food analogy, presumably, some food manufacturers back when before it was mandatory didn't want to have the disclosure because maybe of IP concerns, maybe they have the actual secret ingredients in their minds, or maybe others were too embarrassed to say that they just put regular flour in there and not just some kind of a magic grown in high mountain flour. So I don't know, my analogy is a little silly.

Third quality horse meat, you know, you never know what's in there.

Exactly. So thank you. So how do you deal with that? Like, absolutely.

And I think that, you know, the analogy is a good one because it does extend to, you know, reasoning like this and questions like this. And you're right, in the late '30s in the States, you know, it was the first legislation around ingredients labels for packaged foods. And over the '40s and the '50s, food manufacturers began to put ingredients labels on the foods. And there was some initial resistance. There were, you know, food manufacturers saying, well, this is my secret sauce. I can't reveal this stuff. And there were others saying, well, gosh, you know, I would feel a little embarrassed to tell people what went into my peanut butter because there's so much coal tar and quinine or whatever in there. And—

Ooh.

And I think, I mean, part of what makes SBOMs particularly current in the United States at the moment is some rulemaking that the OMB, the White House Office of Management and Budget, has done around federal, US federal government use of software. And there's an SBOM requirement as part of the executive order. And part of the policy goal of that requirement is exactly to push these kinds of considerations upstream. And there are organizations who are now already thinking, well, If we're going to have to disclose what goes into our software, we'd better get selective and careful and deliberate about selecting our dependencies. And that's a result that's by design of the policy.

That's a good thing. So I want to torture this analogy a little bit. And what's important about your food is not just what goes into it, but how it's processed and prepared. Is there like an SBOM equivalent for, you didn't build it on a server under somebody's desk that's been backdoored with 18 different government agencies' code?

I love that question. And yes, absolutely. And so, if you think of SBOM as, hey, it's a list of ingredients, at a very high level, what went into the making of this thing, there's a separate framework called SALSA, Security Levels for Software Artifacts.

Oh.

And you can find this at slsa.dev. And the SALSA specification defines a metadata format called provenance, and provenance speaks exactly to what you're talking to. And so, if SBOM is the list of ingredients, Provenance is the recipe and the food handling safety standards and what machine was used to make this peanut butter and so on. And so, you get a much better sense of potential for internal tampering and contaminants and what machinery was used to make the food, and do I trust that machinery? And so, salsa provenance speaks to precisely that set of concerns for software and what recipe was followed, what machinery was used to build the software. What precise ratio were the ingredients specified in, and so on. And so I think the analogy stands up even to that torturing. You'll have to work harder to—

That's awesome. But so go ahead, Anton.

That is actually kind of fun because I feel like this details— I almost want to ask about what if the cook wasn't competent? What if he was a jail cook before or something? I don't know. Let's just ignore this part.

I don't think we need to get into bathtub gin on the podcast.

No, I mean, I love that part and I can speak to it. Just very quickly on that. I mean, I think part of the theme we're seeing in software supply chain is one of the most elementary things about supply chains in general, and the manufacturing industry has learned this, is traceability of ingredients up and down the supply chain. And if I'm driving a car, those wheels came from this manufacturer who used steel from this foundry, who got their raw materials from this mine, and so on. And being able to trace that lineage and that provenance is super important. And so, The same work is going into the software supply chain to try and instrument it with traceability metadata left to right. And what you're speaking to there is the idea of developer or maintainer reputation, let's say, for open source.

Right.

And so, using a given open source artifact, I may want to know the ingredients label, I may want to know the source of provenance, the machinery which built this artifact. I also may want to know some information about the cooks. I may also want to know about how did this project respond the last time a vulnerability was found, and what's their mean time times resolution, and so on. And so, that set of metadata, you know, we're in process, I would say, of instrumenting the software supply chain left to right with exactly this type of metadata.

That is actually really cool. That is really cool.

It's really cool. We're all delighted about this, but I wanna pick at something you said there. And I know Anton is gonna be mad that I'm hopping all over our questions doc with this, but what you just said about open source, I think I wanna pick at that and say, like, we're talking about all these requirements around labeling and documentation and traceability, It's a lot of work to get that right. Working at Stripe, I saw some of the challenges around compliance and the expense that goes into that. I can see how big organizations like us can afford to put all this process in place and afford to make all this information available. What's going to happen to open-source projects? Are they going to have all this overhead and make it even less fun to be an open-source maintainer?

It's a really good question. So, I mean, I would have 2 parts to my answer. I mean, first of all, I would note that Salsa itself and the idea of Salsa provenance Salsa is an open source initiative itself. It sits within the OpenSSL, it's part of the Linux Foundation. And so Salsa itself is an open source initiative. And we're beginning to see even some package ecosystems in open source implement Salsa end-to-end. And so start to build Salsa provenance into the supply chain story around open source. I think that there's, you know, zooming out, there's the larger picture here around how we think about accountability in open source, and particularly enterprises consuming and relying on open source. are increasingly looking for accountable entities. Who can I hold responsible, or who can assure me, is there an entity that I can trust who will stand behind this open source offering? And so Red Hat has been in this business for a long time, and Google itself, we have a product called Assured Open Source, which is, you can think of it in these terms, that this is open source that is curated by Google, selected by Google, built on reliable and certified and standardized Google-built infrastructure with the Google imprimatur stamped on this open source offering. And so the product itself is the open source offering, but Google puts its hand up as an accountable party as part of the go-to-market of that. And so I think that there's what you point to, I think, is potentially an accountability gap in terms of where the open source ecosystem leaves off today and where enterprises are willing to begin, or increasingly, what enterprises are looking for in terms of accountability and assurances.

And so does this mean that Uncle Sam is going to end up using a bunch more enterprise software instead of saving our tax dollars and using open-source software?

It's a really good question. I don't have a great answer to that right now. I mean, I think, you know, one of the gratifying and encouraging things that I've seen by being close to federal government legislation and policymaking and rulemaking up close is that there is a deep recognition for the value of open source and the overall ecosystem. I mean, there's a real appreciation for the fact that Just in terms of reality, between 85% and 95% of commercial software has some open-source componentry in it.

Only that much?

A world without open source is very, very difficult to imagine. And there's a very stark recognition of this fact in the government that you can't get very, very far in building modern software without relying on open source in some form. And so I'm pleased and encouraged to see that recognition in the federal government.

Let me try to ramp up the temperature even more a little bit because some of the SBOM conversations that I've seen, some of the software supply chain security conversations, seem to be circling around the question of software reliability. It's kind of like a first step towards that or has some kind of connection inherent or otherwise to that. So, can you explain that somehow? Like, is there a relationship between SBOM and software reliability? Where if you write the software and if it causes problems, then you can suffer deservedly or not, separate story.

Liability is a large topic on its own. I'd speak in general terms about it that looking at particularly the US federal government and the executive order, and the executive order itself speaks to using, quote, the purchasing power of the federal government to induce more secure activities in the supply chain. I definitely see the beginnings of this observation that the security of software, or rather the cost of insecure software, has in large part been borne by consumers of the software to date. And I think that there is directionally an effort to pick up those costs from the consumer and begin to drop more of them on the producer and say, if you're producing software, you should bear some of the costs of producing secure software. And so I see as a trend— I do see that and I see legislation as one of the few mechanisms beyond the free market that can just unilaterally decide to go, we're going to pick up these externalized costs and actually require them to be internalized today. And so trend-wise, directionally, I see that happening. I think liability is a super tricky subject and I think it's too early to see frankly how that's going to play out right now. But I think shifting costs from consumer to producer, is definitely part of the trend I see.

I think that's a really nuanced and thoughtful take on the role of government when it comes to economies like this, and exactly the kind of place where my political science brain says, yes, this is a good use of government to make a market work better where the market itself was unable to make it work correctly. So that's great.

And what you just said, I think, is absolutely the nub of it, that these are not outcomes that the free market on its own would necessarily drive towards. And so I think with the right policy touches and right policy design, you can reshape the incentives and reshape the economy here to the better greater good. Whereas market forces may drive you to local optimization, I think it's for government broadly that has the ability to optimize the ecosystem as a whole. And so far, as I say, so far I've been relatively encouraged by the fairly progressive and enlightened thinking that I've seen in the US federal government along those lines.

Yes, I cannot agree more about the difference between local optima and global optima and the role of government in regulating markets towards that. I mean, really, markets can't exist without government in the first place because you'll just get a bunch of scammers scamming scammers. And a market is a miserable place without, say, regulated weights and measures.

You mean like our favorite topic?

Like crypto. I like crypto. So, so I want to, I want to maybe get out of my, uh, now decade-old political science degree and think a little bit more locally and shift gears a little bit to what's Google doing to get ready for the EO internally? I know you have written a really clear doc on this. And how are we using internally SBOMs and related tools?

Yeah, absolutely. And so I think about the executive order, there are 2 major pieces for organizations to ponder. The first one is that the federal government is going to require software in use by the federal government and federal government agencies The software should be built in conformance with a NIST standard called the SSDF, the Secure Software Development Framework, which is a large set of how you should approach through various phases of software development, how you should make that process secure. The second part of the executive order I think is significant is this SBOM requirement. And at Google, we're a relatively mature software building organization. And so, looking at the SSDF, It's a very rational set of practices which are recommended, and it maps very, very well into Google's internal standards. And so, Google's own internal baseline for software security compliance and so on is already very close to parity, or in some cases beyond where the SSDF is.

Oh, that's great.

I think with SBOM specifically, again, there's large parts of Google, particularly in Google's monorepo, where, you know, we've been, quote unquote, living the SBOM lifestyle for quite a while in terms of, we've been making sure that artifacts are tracked. We have an audit trail left to right. We have strong systems for binary provenance. So, a given artifact that's running in production as a binary, we can trace that with cryptographic verifiability back to, you know, the exact set of source commits, which made that binary. And these are systems which Google has already had internally for a while.

I have a really funny story about how traceable everything is at Google.

Oh yeah.

Early on in my time working on one of my log teams, we were sitting around, we had just launched something, we're all high-fiving about how great it is that we finally got this thing live. And my tech lead gets a ping from a traffic SRE, which is not somebody you want to get a ping from.

Right.

The traffic SRE says, hey, do you own this job? And Chris says, yeah, that's my job. What'd we do? And the SRE says something basically, without giving too much away, to the effect of, do you mean to be saturating an entire fiber optic cable between Asia and the United States? And my tech lead said, oh no, I didn't mean to be. And the SRE, and this blows me away 5 years later, no big deal, just deploy better next week. It was incredible.

That's amazing. Yeah, I love that story. And that speaks to a lot of Google engineering culture encapsulated in that one story, actually.

Yes. But the trace that, you know, this random Borg binary out of all the Borg binaries could be instantly traced back to from whence it came, really speaks to Google.

Absolutely. And so, I think, you know, for SBOMs, and again, you know, I've been driving a lot of the work internally to make sure we have, you know, all of the tooling and processes necessary to produce software bills of materials for Google first-party software, because, you know, we have a lot of this traceability infrastructure and observability of our software development lifecycle internally. In many cases, it's really just a matter of emitting the SBOM document in the right format at the end of the day.

There's 2 standards for that, right? 2 document standards, and they don't agree yet?

There's a couple of prevailing standards. There's CycloneDX, and there's SPDX, and they have different provenances and lineage and different history. And I think people should take a look at them. There's varying support in the tooling ecosystem, there's varying support in the consumption ecosystem. And I think on that latter part, I think a big gap in the ecosystem generally is around Okay, if we have all of these companies and all these projects and all these software producers building SBOMs and producing SBOM documents, how do we reason about them at scale? And so when the federal government wakes up in the middle of June to find an inbox with a quarter of a million SBOMs in, what do they do? And I think that there is a tooling gap to be closed there about how do we reason about all of this metadata at scale? Let's say we do have provenance and ingredients and information about the chef and how late he was for work last week and everything about this jar of peanut butter. Now I want to reason about an entire grocery store of packaged food. How do I do that? And I think that's a really interesting problem that there isn't a great general solution for today.

That's a really interesting problem of reason about the whole grocery store. Oh, gosh, I hadn't even thought about that.

Right, because ultimately we are not doing it just for fun. We are doing it to eventually improve, right? Well, I don't think these are very fun artifacts to make, but So, I wanted to take the conversation not in the direction of like, hey, we're going to be transparent. We're going to be having this machine reasonable thing and it's going to be epic, but stuff should be better security-wise, right? It shouldn't just be transparent. So, how is this all improving our food, to ride on the same analogy, rather than just making us aware of what we eat? So, whether it's executive order or our own efforts or like, Where are we heading in the next 5 years in terms of improving supply chain security rather than just knowing what's in it?

Improving, you can't get started on improving until you've made some traction with the knowing. Knowing has to be the first step. If I want to be very selective about what dependencies I'm relying on in my software, and I want to do a better job of selecting secure ones, I have to first know what are the set of dependencies I'm relying on today. I suppose certainly, have to have some sense of inventory in order to then whittle it down over time. And so knowing, I think, is the first step. I would say as well that in requiring SBOMs to be produced, there's a certain amount of operational discipline you have to have internally in order to build an SBOM. And so by requiring companies to build SBOMs, you are transitively almost requiring them to develop some operational discipline, which itself will be beneficial for their supply chain. And I think that what we'll see over time, SBOMs is just the very, very beginning. I think that the executive order is requiring the software industry to build end-to-end traceability at the SBOM level and build what I'm thinking of as a metadata bus, essentially. So, as artifacts move through the supply chain, they're going to have metadata attached. And then step 1 is that metadata is going to be SBOM. Step 2 is we're going to have all kinds of other metadata on that same bus. But once we've built the bus, we can use it for all kinds of different purposes, including provenance and develop a reputation and first-party threat intelligence and VEX statements, which we haven't even touched on. But I think you're right in the characterization of the question here is, gosh, SBOMs only get us so far, to which my answer is absolutely. SBOMs are really very much just the beginning. And I think of, I mean, to Tim's point—

There's a plumbing behind improvements. Unless you build the plumbing, you can't improve. But now we are building plumbing. I'm away from food and energy.

I don't know about that one.

Okay, fine. Maybe You're building infrastructure for improvement. SBOM itself, transparency wouldn't magically improve things, but it would allow you to track whether you are improving.

Right. Yeah, I think that's true. And I think it's the very beginning of this. I mean, the executive order is the US federal government with a particular form of requirement for SBOMs in particular circumstances. But I think that evolutionarily, you can expect what the federal government requires today is what state government will require tomorrow and municipal government and then large enterprise. And there's already the EU has the cyber resilience Act, which speaks about SBOMs as well. And so I think just generally, we're seeing, you know, gathering momentum in the industry to better improve our supply chains, to be more careful, have more discipline around them, have more observability and traceability around them. SBOMs is just the very, very beginning of this journey. And, you know, you can think of it as, you know, hey, it's 1942. And, you know, the first candy bars with ingredients labels are just hitting the market. And it took about I think after the legislation came in in the early '40s, it was about 20 years until half the US food supply was covered by federal regulations. I mean, it was a slow-moving thing. And so, you know, to Tim's poli sci background, part of the general challenge here is just change management in general. Like, how do we, how do we bring change to an industry at large, at scale?

Yeah.

And I think SBOM seems to be a reasonable thread to pull on. But it's just the very first thread and we should expect to find an entire sweater in our lap before long.

One thing that occurs to me as we've been talking about this is it never occurred to me not to trust a food label. How do I trust an S-bomb? And now I'm going to go worry about food labels.

Oh, wait a second. You're triggering my paranoia. Aren't they supposed to be correct?

What if the S-bombs are wrong?

Why would people lie? Well, people would, I guess, forget that they included a lie.

People lie all the time.

Okay, fine.

I don't want to say that there's rat droppings in my cereal box, but of course there probably are. So how do we trust them?

As a general point, I would say, particularly in the context of the executive order, lying to the federal government is an extremely bad idea just to begin with. And so I think that you'd be much better just not providing an SBOM or providing a remediation plan for how you'll develop an SBOM rather than lying about one. What gets interesting when you go back to things like salsa provenance is salsa provenance, the mechanism by which that's produced is there is a trusted entity, we call it a trusted builder at the core of this process. And I anchor my trust in the builder. And the builder will then say, well, I saw these ingredients come in the front door, and I saw this binary go out the back door. So I'm going to say, there can be nothing else, because I'm hermetic as a builder. I live in an ephemeral and isolated environment. I can reason very completely about The set of things I see coming in and the thing going out, and they can then sign a document saying, I saw this happen. And if I have trust in the builder, if I anchor my trust in Google Cloud Build or in GitHub Actions, whatever is doing the building, I can then make certain inferences about to what extent can I trust the provenance. But it's a really good question.

But think about how radically different what you just described is from how I imagine most software still today gets Like, what you just described is a spaceship factory in orbit producing the USS Enterprise. Most people are doing it on their developer's laptop, which hasn't been installed critical operating system patches.

I think that's very true. And so, I mean, again, with Salsa as a great example, Salsa itself as a framework comes in levels. So, level 1, level 2, level 3, and so on. And the idea of this framework is precisely that, it's a journey. And no matter where you are, on your software production journey, you're going to correspond to some level of Salsa. And wherever you are on the ladder, you can reach the rung that's above you, and then above you, and so on. And so, Salsa itself has the flavor of a transformation framework that it meets you where you are, and then guides you up from there. But I think, coming back to the change management problem, how do you get from the Mac Mini in the closet to assembling the Starship Enterprise into orbit? That, again, is a change management problem. And so, you have to very carefully design these frameworks with recognition of where we are as an industry today. Totally valid observation.

That makes a lot of sense. So we're at time, listeners. Thank you for joining us. Do you have, Isaac, recommended reading and one tip for improving your software supply chain security?

So recommended reading, look, I'm just going to talk my book again. salsa.dev is a great place to start learning. There are some blog posts on the Salsa blog about how Salsa compares with SBOMs, how you should think about Salsa in the context of the executive order and NIST SSDF. So it's the Salsa blog and salsa.dev, again, S-L-S-A. is a great place to just kind of begin your reading journey. I think in terms of where to get started, the thing that I continually tell people is, start getting a sense of what your dependencies are, if you don't already know. I was at Cloud Native SecurityCon a couple of weeks ago, and, you know, on a panel, and so, you know, the moderator said, hey, in the room, hands up if you know what all your open-source dependencies are. And not a single hand went up. And then the question was, hands up if you feel like you ought to know, and every hand went up. And I think closing that gap is a great place to start. Get an assessment, get a feeling for where are my dependencies today. Build that sense because knowing is the very first step on this journey. Once you know, you can then begin to measure your risk, reduce it over time, reason about it, and so on. But you have to start with knowing.

I always tell customers, you can't manage what you can't measure. You certainly can't secure what you can't see.

Exactly.

So getting visibility on your open source components, that makes a ton of sense.

Totally.

Isaac, thank you so much for joining us today. Listeners, there's going to be a lot of great links in the show notes, including links to some of our older episodes, like Tim Dirks talking about building codes, Eric and Aparna talking about Salsa last summer. Lots of good stuff in here. Isaac, can't thank you enough for joining us.

Thank you so much, Tim and Anton. Great to be here.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikin and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.