EP226 AI Supply Chain Security: Old Lessons, New Poisons, and Agentic Dreams

Security running.

Hi there. Welcome to Cloud Security Podcast by Google. Thanks for joining us today. Your host here, as ever, myself, Tim Peacock, PM for Threat Detection and UEBA and some other stuff in Google SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find the show wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want us delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with your hosts, and the rest of our Cloud Security Podcast listeners all on the Google Cloud Security Community page. Anton, we've got a great episode today with Christine, so let's get straight into it. Christine, thank you so much for joining us. Our guest today, Christine Sizemore, Cloud Security Architect here at Google Cloud. Christine, could you walk us through something very exciting and new, an AI software supply chain. And is that kind of the same as an SBOM, or, you know, the components that go into that, or, you know, what's different compared to a traditional software supply chain here?

Yeah, thank you so much, Tim and Anton, for having me today. I'm super excited to be here and to be talking about AI supply chain. But to your question, I think before we get into like the components of AI supply chain, It would be good to talk about how fragile supply chains are and how vulnerable they can be. One example your listeners may be familiar with is that Suez Canal disaster that happened a couple years ago where that one boat went in a diagonal, blocked the whole canal. There was like 400 ships waiting. It had like global shortages impacts for months. Do you all remember that one?

Yeah, yeah, yeah. It was a big boat, the Ever Blocked or something.

Really good pictures also. Stuck in a canal. Yeah.

Yeah.

And I think that just does a really great job of showing how fragile supply chains can be and how they all need to be working together in order to have a seamless experience for the customer. But back to your question around what components are in the AI supply chain, I have another analogy for you. This one's with cars. You can think about with cars, you have the raw materials like steel. And with AI models, you have things like datasets. Then what you're going to do after that is you're going to assemble and make the car in the factory. And with AI models, what you're going to do is you're going to use training frameworks like TensorFlow and PyTorch. And kind of relating to software supply chain, those training frameworks that I listed are just software libraries, and you need to make sure that those libraries don't have any vulnerabilities and are coming from trusted sources. Then if you think about that car example, we've got it at the factory, you're now going to ship it off to the dealerships. You can think about with AI, I like to think about this as model serialization. And if you're not familiar with model serialization or code serialization, what that is, is making the model portable.

Got it.

So you can take it off your AI training infrastructure and, like, take it to the quote-unquote dealerships of, like, Google Cloud, Azure, AWS, or on-prem. And then just like with traditional supply chain security, you want to make sure that your artifacts that get created during that, your model artifacts, just like container artifacts, are have artifact integrity and aren't being tampered with. And the last part I have for you all is, you know, if you think about the car, it's now at the dealership, it's going to be sold to the public. This kind of reminds me of things like the Kia Boys. Do you remember that trend from a couple of years ago?

No. What trend is this? You've reached our cultural depth.

Okay. We reached our cultural depth.

Yeah. So I think in the pandemic it was really easy to steal Kia cars. They had like some—

they didn't do enough work. Oh yeah, yeah, yeah. With like the USB stick.

Yes. Yeah. And it became kind of a trend on TikTok. To steal these cars. And this kind of reminds me of prompt injections. If you don't have enough guardrails around your LLM, then you're going to be susceptible to a lot of prompt injection attacks. So that's kind of how I see how those different supply chains relate to AI supply chain. Wow.

I never would've thought we would go from the Ever Given to boys committing grand theft auto on TikTok for the lols, but here we are.

I mean, while we are Talking about this, I'm getting this slightly uncanny feeling that when I think of AI security issues— let's do a 3:00 AM test. Somebody calls me up at 3:00 AM and says—

Ah, 3:00 AM test. Here we go.

Anton, what's the top AI security issue that humanity faces today? And by the way, no, it's not robotic rebellion or anything. But the thing is, I won't think of a supply chain. I would think maybe data theft, data poison— maybe I would think of not poisoning. Maybe I would think about prompt injection. So among other AI security issues, why does this matter? Like, why is AI supply chain more important than, say, putting a filter up to stop prompt injection or training models to avoid it or something else?

Yeah, I would consider it to all be a part of the supply chain. Like, you need to have your supply chain in order to have a, I guess, safe, quote unquote, safe model. You need to make sure that your datasets are not being tampered with to avoid, as you've talked about, data poisoning, where they could be manipulated and have the LLM just saying misinformation. You also want to prevent things like model poisoning, where the model— there's actually one example. It was by a security research team. They called it Sleepy Pickle. And what they did is they altered the weights for ChatGPT Medical. And how they altered it was if you ask the model, hey, I have a cold, I'm not feeling well, it would say that you need to drink bleach, which we all know you should never, ever do. But—

Wait, I thought that was how you treated COVID.

Those components are so critical a part of the supply chain. And then at the very end, you have prompt injections as well. trying to mitigate against that. So, I think as a whole, supply chain really ties in all of those AI security risks together, and making sure you get each component correct really helps mitigate some of those other risks you may be thinking about.

Some of the stuff is really runtime, though. I'm trying to gently push back on this because, say, prompt injection isn't, in my mind at least, not connected to supply chain that much, right? Is it?

I think that with prompt injections, if let's say your data has been poisoned in a certain way by a malicious actor and they know about that, then they could kind of provoke that model through a prompt injection. So I think it could tie in with that way. Okay.

Yeah, that's fair. That's fair.

That's fair. Yeah.

So on this show, Anton likes to have the 3 AM test. I like to say that history doesn't repeat itself, but it does rhyme. What are the rhyming practices of previous supply chain security that we need to be thinking about most importantly for AI?

Yeah, I really like that phrase, because we are really lucky to have a lot of the lessons learned from traditional supply chain that can really extend into AI supply chain. When we think about it as an industry, we all kind of have an idea of how code should go to production. It should have like a CD/CD pipeline, provenance, attestations, logging and monitoring. And when you think about how does a model go to production, we haven't as an industry established that. And part of that is because models are everywhere. There's so many different ways to consume them. And so I think some of the lessons learned that are being taken into AI supply chain from software supply chain are things like provenance. So provenance, making sure you have essentially a recipe for how your model was made. You can have things like data provenance. So understanding where your data is coming from, having data lineage to know any modifications to your datasets that could potentially be malicious. Then you have model provenance and tools like Weights and Biases, which will record every, let's say, experiment that's been done and ran with your LLM model and the parameters that were used, who did it, logging and monitoring to really understand everything that went into that version of the model. And then other things that you could take from traditional supply chain, like I talked about artifact integrity. So with software supply chain, we— when Docker first came out, People would just go to Docker and download the container files. And kind of with AI and LLMs, people go to Hugging Face and just download those models. And we know better now to, instead of just like download any model you see, to verify where those model artifacts are coming from. You know, get the CAR facts, the CAR history.

You know what's so funny to me about that? I grew up on the internet and even as a like 9-year-old, I knew it was a dumb idea to download that random like fonts file or that emoji pack for AIM. It blows my mind that there are professionals paid to do these jobs who are just like, oh yeah, I'll just download that thing.

Yep. Yeah.

Yes.

Yeah.

Like, how is our industry still in this mode?

I think that we're going to go far into the— I don't know what direction, but I think that my suspicion is that it's high trust.

Is this too depressing?

No, no, no. It's the opposite. We are a high trust society. We still are, despite the political events and all that. And, you know, I don't want to quote Bruce Schneier too much, but he had a book about trust and societal trust specifically. And to me, this, like, very much like if you are grown in a high-trust society, you are— it's sort of okay as like, I'm going to go into a site on the internet and get some code. Like, I think it overrides your IT intuition that you are born and raised in a high-trust society. I think that's my suspicion.

I thought I had grown up in a high-trust society. And then I was in Singapore 2 weeks ago, and I saw people reserving tables at the Hawker Center By leaving their car keys on the table.

Yep, yep, yeah, I know it's so—

yeah, yeah, just like, I'm just gonna leave my keys here, walk away in public, and I'm sure not only will I be able to sit there when I get back, my car didn't leave.

My car would be there.

Yeah, yeah, no porch pirates in Singapore.

But can you imagine?

I also think, to your point, it's a high-trust society, but also there's like a business need to move as fast as possible. Not always move as fast as possible, but there is always a business need to drive and move forward. And there's, like, you know, bonuses associated with certain IT projects. And so, I think you're also fighting that, as well. It's kind of the battle that security always goes through, like getting business to be done and then doing it securely.

Yeah. But it's also— I want to steer back briefly to trust because it also seems like, one, it takes us 10 years to learn that maybe don't download executable files, you know, free software, shareware, blah, blah, blah. Sorry, I'm dating myself. to the '90s or whatever. And then suddenly we fought this battle, more or less won, and then it's, don't click on the links. We fight this battle, we kind of haven't won it yet really, and now suddenly, don't download models. So it's interesting how this really rhymes. It just rhymes with a freaking marching band. It loudly rhymes. I don't know, my metaphors suck.

Do marching bands rhyme? Listeners, you heard it here first. Marching bands known for rhyming.

Yes. But it's like now suddenly, why is downloading random code from GitHub not secure, but a model from Hugging Face is secure? That's not logical at all. It's the same public website, or different public website. We're going to have supply chain fun for years to come.

Yes. Fun is one word for it.

Yeah.

So, as far as to look at organizations, and yeah, we sort of touched on the organizational pitfalls, but if I am maybe a CISO, he's part of his team, and I've heard your webinar or I've read the paper about the AI supply chain and I want to do it better, or maybe even first not do it worse first. So, what are some of the organizational pitfalls to avoid? Like, are there any specific, you know, themes that are associated with irresponsible and wrong things, like downloading models from a random site?

So what's the bad smell of naughty AI?

What's the bad smell?

How do I smell bad AI?

Oh my gosh. I think that from working with my customers, one of the biggest pitfalls I see is silos. Organizations— I know it's very tough and silos happen when you have a large organization. But when I see companies that either their security team is too rigid, not willing to accept risk, the AI team goes ahead and does shadow AI. Or you have the AI team that will push ahead and then pull in security last minute. And then security has to do a, like, last-minute security review with only like a month until this thing goes into production. And so I— those scenarios are both not good scenarios for AI supply chain. And I would say trying to work on those silos, bring in security early, and then for security to understand and kind of centralize on what are the risks you're going to take with these models, which data you're going to allow for models to interact with, which users, and really assessing the use case by use case on the risks you're going to take. I think those are some of the pitfalls. I think another one is also lack of monitoring and visibility. With supply chain, it's so critical to have an inventory of your models. And that's really hard to do. There are a lot of ways to consume them. But I think one of the ways to kind of avoid some of these challenges is to get a good understanding of what all models are in your environment.

So I guess I hear that. And it sounds to me like those would smell bad for anything, not just smell bad for AI.

Exactly. Thank you. The same comment I wanted to make. It sounds like we are talking about software or some other code or binaries or some other stuff like this.

Or honestly, any process issue like silos, rigid security teams, lack of visibility and monitoring. That's bad for literally everything.

So being invited late comes up, right? Like in the Brockman Orr episode, right? We had this brief debate about how security has been invited late. So '90s, so 2000s, so 2010s, so 2020s. And like, that's still the story of our life, right?

Yeah, I think it still applies as well to AI supply chain just because these are things that we need to get and work on and nail down, like bringing security early, following security best practices. I guess another one, I think for AI supply chain in general, the inventory one does still apply, knowing which models you have and which versions of them. Which datasets you're using. That can also really help to ensure that you're understanding your supply chain and not introducing malicious actors. I think other things too would be testing. There's this thing called drift detection. That's also pretty general, but making sure that you're understanding the drift between your desired response you want the model to have and then the actual response the model has. That way you can see if something malicious is going on, like all of a sudden your model's saying that you need to drink bleach.

This sounds very— it sounds very not magical. It sounds very normal, mundane. I'm not at all contesting this advice, by the way. It all makes sense, but it makes sense too much. I'm sorry, your common sense is too strong. Please turn it down. So no, that's what he did. Is there something magical about— there's this magic wand, you speak into it, you prompt it, and then suddenly magic happens? Is there anything else but do IT well?

I think that, yeah, leveraging, like, model provenance tools. I think, yeah, I think I will stand by trying to make sure you have good visibility, bringing in security, leveraging those tools I brought up earlier, artifact vulnerability scanning. And I think we need to nail these best practices down. I know it's not magical, it's not exciting. This isn't, like, a nice little Disney story, but These are things that we need to nail down in order to avoid some of those threats I was talking about earlier.

That all makes altogether too much sense. Can I ask, Anton, can I ask the off-the-wall question since we're making way too much sense?

Of course, we have the walls, right?

Yes, okay, great. So Christine, I feel like we hear all about this agentic AI security. Why can't I just ask the AI to secure itself? Why can't I say, hey AI, introspect how you were created, and figure out what was wrong. And to that end, why can't I ask the AI, you know, hey AI, watch out for prompt injection attacks and don't respond to them?

Gotcha. Yeah, I can understand the sentiment a lot. If you think about even before AI came out, there was just way too many findings and alerts for a SOC team to manage, like a human SOC team. And so LLMs are definitely coming in to help with that and analyze and be able to help bring together some of the most important findings. But I think for right now, just with some of the threats I talked about with models, we cannot rely on just agents to do our security because of things like data poisoning, model poisoning. I think maybe later, as we start to vet out these solutions— I think this year is really kind of where agents are going mainstream.

Hmm.

So I think maybe in maybe 5 to 10 years, maybe we could have it just have an agent do kind of like a— Kind of like a test that you— like an automated test that you would have in software supply chain to test your code. You could have like an AI, hey, can you do this prompt injection red teaming test? I think that would be a great idea, but I think you can't at the moment just rely on agents alone.

I mean, it kind of makes me nervous that you say that. And here's the— it's a really weird reason. I think what you said makes total sense and it's probably correct to the best of our knowledge. But because people show up at our door and probably at your door as well and say, Can I please get a scanner that I aim at the model and scan it and it tells me if there's badness in there? Like, the short answer is no. Guess what the long answer is?

No.

No.

Yeah, after a while.

But do you expect this to happen in the future, that you would have the AI-based, fully automatic badness scanner/detector?

You know, I was trying to think about About what things have been automated completely. And like, even with cars, we're still not fully there. They're not fully used with automated self-driving. And I think that I understand what you're saying, where the scanners, like, soon the bad actor is just going to know what they're scanning for, and then you're gonna have to alter it. I think, I don't know, but I think maybe in 10 years, once we have agents, you know, do the datasets, train the model, do the prompt injections, maybe we could get there. I know that's not a great answer. That's a maybe. But yeah.

I think maybes are great.

I mean, it's a good answer because I take a maybe over a crazy enthusiastic yes, or we'll have AGI and we'll tell it to solve all problems and say, AGI, solve all problems.

Yeah.

Don't just solve security. Yeah. Well, we know how that ends based on sci-fi, but of course, that's probably a bad guide for future.

It depends which sci-fi universe. Culture series, I'll take that solution.

Well, are we not renaming it into sci-fi In the Cloud podcast or Sci-Fi with AI or something. Okay. So, I really digress. Practical question. So, if you're giving advice to somebody who runs security at an organization, whether it's a bank or a manufacturer, and you did give your passionate speech about AI supply chain and they got it, you see their eyes light up and they're like, oh, yeah, that's important. Are there 3 things they should do now? And please don't include one, do IT well. Because sure, but is there anything practical and more specific to a domain of supply chain for AI they should do?

Yeah, I think that trying to get, one, an inventory of the models that they're using is good. And then second, making sure that they have some vulnerability scanning, making sure that they're using things like Protect AI or Hidden Layer, making sure that the model artifacts that they are using don't have things like pickle files or safe models, instead using SafeTensors. And then that testing I said earlier with drift testing. And then another area kind of is when you think about developers, we had like you have scanners on their Visual Studio, for example. With data scientists, you want to make sure that you have scanners on their notebooks that they're interfacing so they're not downloading malicious libraries. I think those are some great kind of easy picking fruits to take from this podcast. Some low-hanging fruits.

Nice. I love low-hanging fruit personally. We were just talking about this the other day. I once had a CEO who was not from America and had never seen pumpkins grown say, we're hitting our heads in the pumpkin patch, the fruit is hanging so low.

Oh my gosh, yeah.

Which of course for me brought to mind the image that like the organization had like passed out, fallen over, and was rolling around on the ground. It's not quite what he meant, I think. Christine, I've enjoyed this conversation tremendously. I'm sure our listeners have as well. I hate to do this, but it's time to ask our traditional closing questions. Do you have one tip to help people improve their AI supply chain security beyond the 3 you just gave, perhaps? And 2, do you have recommended reading? And that recommended reading could be anything from Mary Oliver to Mary Shelley to the Culture Series, but not Anton's blog.

Okay. There's a great research paper on AI supply chain security by Google, the Google Research that is put out. And Anton does have a blog that kind of summarizes that if you want to take a look at it. There's also some readings from Protect.ai, their insights database, that outlines a couple of AI supply chain threats that give a nice overview of them. And then I think for the tip, definitely would be getting an inventory of what models are being used in your environment, just so you understand and can then evaluate their supply chain.

I think inventory is such good advice.

I know it's not magical.

Yes.

I know it's not magical.

It is a little bit in the Dwight D. Well.

It is magical. So much good falls out of having a good inventory.

Okay, it's like one of those load-bearing pieces of advice, right? A lot of stuff just sits on that, and if you don't do that, a lot of stuff crashes on you.

Yeah.

But let me ask you an untraditional closing question. No, I don't have it prepared. How did you end up doing all this? Tell us a bit about your journey.

Oh yeah, yeah.

Here's the best piece of story. Ideally, keep it to 1 minute. I'm joking, of course. Keep as long as you want, but how did you end up doing it?

Yeah. Went to school at Washington State University, go Cougs, and I studied management information systems and they had a security class in there. And when I learned about cybersecurity, this like big sister instinct that I have just really kicked in and I was like, I really wanna help people. Like tech can be very overwhelming for some folks and it wasn't overwhelming for me. And I was like, this is where I can actually help people because when I see blood, when I wanted to be a doctor, I got too queasy. I couldn't do that. So I was like, I could do this.

Yeah.

And then I started at Dell and did, like, infrastructure work. And then I came here to Google and was always volunteering for cybersecurity challenges and projects and doing certs. That's kind of how it really, really goes back to that cybersecurity class. And then just keep putting my hand up for security projects.

I love that answer.

Oh, thanks.

It does make sense. Yes. And I think that's— you are, in that sense, you didn't study art or political science or something. You actually studied information systems, which makes you— yeah, I know, Tim, right? Well, you didn't study physics either, so.

No, that's true. Thank God I didn't write a PhD thesis about— what was it? Down quarks?

Yes.

Well, you know, Anton, we actually both had fun pieces of swag in this episode. I had our mug and you have quite the sweater on. Do you want to tell us about your sweater?

Yes, it's the belated Christmas sweater and I wanted to make a joke, which was not very tasteful, that I bought it for $32 billion. But no, I actually got it as a Christmas gift. I got it as a Christmas gift. I did not pay for that sweater. Waffles, by the way, seemed to be thinking I'm not telling the truth, but I did not buy it for $32 billion. I got it as a gift. I think he says yes.

Waffles doesn't like your joke. Waffles apparently doesn't understand balance sheet math. All right. Well, listeners, with a late introduction of my dog off camera, Christine, thank you so much for joining us today. Anton, thank you for wearing By far the ugliest thing we've had on the show so far. With that, listeners, thanks so much for joining us today.

Awesome. Thank you.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Cevakin and Tim Pico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.