EP144 LLMs: A Double-Edged Sword for Cloud Security? Weighing the Benefits and Risks of Large Language Models

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Tim Peacock, the Senior PM for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button, ring the bell, hit the like, whatever it is in your podcasting app of choice. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on our LinkedIn page. Anton, we are talking about LLMs today, and I have to say, at the end of the episode, you had what I think is indeed an epiphany and not That's correct.

And I think that this episode is also a really good balance of depth and lightness. Like, it's both fun and deep, which is really—

At the same time.

Yes, at the same time. And it's actually quite difficult to get that. And I think with our guest, we did get that. It's useful, it's fun, it does go in really deep. 2 acronyms we both had to look up.

Yes, yes. We all learned a lot during this episode, which I think is really cool. I think for you and me to learn things is definitely a mark of a good episode, and for it to be so fun is also a mark of a good episode. So maybe without any further ado, I couldn't be happier about this week's guest. Let's turn things over. Today is an incredibly special day for me as a podcast host because I finally get to welcome to the show the person who said, yeah, sure, go for it. What's the worst that could happen? When I asked, hey, do you mind if I start a podcast? Today I'm delighted to introduce Kathryn Chi, my former manager, today a group product manager and the LLM lead here in Google Cloud Security. Kathryn, I'm so happy to have you here. I've heard a lot about these LLM things. I've heard they can do just about anything or everything, which makes me think they can do nothing. So what is an LLM, and what can they actually do?

Before she answers, I think I want to have the more funny/scary way, which I actually tested on some people on Twitter. When I think about LLM-based chatbots, are we more in the baby AGI camp or more in the glorified autocomplete camp? Please don't say in between.

I am curious how much you tested it on people versus other LLMs on Twitter, but we can get to that some other time.

Hmm.

I think if you force me to pick a side, which I will deeply resent, I think they are probably closer to glorified autocomplete. Like, I do not think that there is true general intelligence However, I think the nuance is that there are probably a great many tasks that we assume require intelligence, and they maybe just require advanced pattern matching and the correct set of abstractions. So LLMs can do all sorts of tasks like that. I think people have seen examples of asking an LLM how to stack 5 objects with different properties and different degrees of squishiness, and the LLM can actually do it.

Mm-hmm.

The thing I would argue is that this is maybe an example of a task where you don't actually need AGI. You need the correct abstractions and some very basic autocomplete functionality. But the abstractions are nuanced and hard, and so it ends up seeming quite intelligent, perhaps more so than it actually is.

Wait, Anton derailed us with his tricky question. I want the basic question. What is an LLM anyway?

An LLM literally stands for a large language model. It is kind of the new hotness, I guess, in technology. But it's basically a very, very, very, very complicated statistical model where you can put language in and it can give you linguistically reasonable responses out. So you can ask an LLM to tell you about Paris, and it will tell you that this is a city in France with these characteristics, and maybe it has the Eiffel Tower and people like to go there on vacation. And then it can do that for a range of more specialised tasks as well.

So it's just statistics that take input and then predict some output that seems reasonable given the input and everything it knows?

At a very, very high level, I think that is a reasonable description. I would also point out that at some point, I don't know, quantum mechanics might also just be statistics.

Anton's the PhD in that.

Yeah, I think we are veering into dangerous territory. I feel like you are underselling it a little bit, but also I feel like you're right about the tasks. I mean, today, short and somewhat funny story from today, or the recording, I needed to— somebody said, we're doing this research report, blah, blah, blah, and we want to interview you. Can you suggest the questions? So my first reaction was to say, Bard, here's the report, write me questions. And then I got the questions and I sent them to that person. And that person is like, wow, these are great.

No joke.

How much time do you spend creating them? And I'm like, Bard did that. And here's the prompt I used. And there was no real interactive prompting. It was just like, This is the prompt. Whoosh. Result.

I think coming out of the— this may be just very, very, very fancy statistics, but there's a bunch of interesting properties that do flow out of LLMs. People talk about this being emergent behaviors. One of the trends you'll hear is—

What does that mean?

Well, one sec. One of the trends you'll hear is that LLMs have sizes measured in parameters. A smaller LLM may be order of magnitude 5 to 10 billion parameters, and then bigger ones may be hundreds of billions. And the trend is that as the number of parameters increases, LLMs start to demonstrate what people call emergent behaviours. There's actually a debate about the extent to which these behaviours are truly emergent, and we could get into that. But basically, if you ask a very large model for things like basic arithmetic or reasoning tasks or kind of summarisation that appears to require a semantic understanding of the data being summarised, the models can do a surprisingly good job at those things. And that starts to get to the excitement around it, which is that there are all of these tasks, particularly ones that require semantic understanding, language skills, or merging and again appearing to understand disparate data sources, that LLMs, especially large ones, are surprisingly good at. And that's driven tons of the market excitement we now see.

So it does sound like some profound truth is hiding behind this. And then I think that—

Or... People want to see profound truth hiding behind it, and the grass is just moving in a way that suggests a leopard, but there's no leopard today.

That's underselling it. Yeah, but I think that a super short summary of this is it's not like the machines are smart, it's people are stupid.

Yeah, I think people are stupid.

I'm going to just take the middle ground. You both can beat me up.

Go ahead.

My own 2 cents is I think that people tend to associate language and intelligence I personally believe they're separate. Like, the example that I have previously told Tim is I believe elephants are intelligent. I believe elephants are probably sentient. They have no language skills. They could not pass the Turing test. I think LLMs are probably coming at something similar from the opposite side, where I think LLMs are certainly not sentient. I think in many ways they are probably less intelligent than elephants, but they have a substantially better understanding and command of language. And I think as humans, we tend to conflate the understanding and command of language with actual intelligence. Like, that was kind of the birth of the Turing test.

Okay. So let's go to our juicy, threaty, hackery details because I think this is fascinating and I feel like we can do it for another 6 hours, especially with Catherine. But—

True.

Let's talk threats. So we do talk about tuning the model. And while this process does look mysterious, there are also security implications of that. So when we think about tuning, are there additional— well, what is tuning and what are the risks connected to maybe data theft and other things? I want to explore that side of the house.

Yeah, totally. So when we think about tuning models, I think there's literal tuning and then there's a slightly broader definition. So the literal definition of tuning a model is basically taking steps that adjust the model's internal weights in order to get the responses to be more in line with what you want. And then models basically go through a couple different steps. There's pre-train, which is when you pour tons and tons of information into the model and it starts to generate its core knowledge and basic, say, English abilities. There's also things like fine-tuning and RLHF, where you probably don't impart additional knowledge upon the model, but you kind of select for the style and type of response you want. Those are both places where you're baking more things into the model. And at a high level, any information you put in in those steps might be regurgitated by the model at a future date. And this often causes great concern for security users, and rightly so, because there's this tension where, you know, you want the model to perhaps operate with your company's own proprietary data, but you don't want that data, like, leaking into the world, understandably so. So how do you think about making the model do what you want without risking regurgitation in undesirable contexts? And the answer is that there's other ways to bend the model to your will. In particular, people will talk about things like retrieval augmented generation, or RAG, or in-context learning. Basically, the same way that a person doesn't have to completely know everything about the world to answer your questions. Like, if I ask Tim, hey Tim, how do I configure a GCP web application firewall to do X and Y? He'll probably be like, I don't know offhand, but I know how to access the docs, and he'll go look it up. He'll read the docs and he'll come back with a reasonable answer.

That's true. Sorry, Emil, I don't know how, but I'd look it up just like that.

You can kind of have models do the same thing. You can provide in the context of a prompt, like, hey model, how do I make web application firewall do this? P.S., the docs are these 3 paragraphs. You can also give models access to what we call tool use. Basically, you can use things like LangChain to make it so that a model can dynamically identify and fetch information it needs.

How does it know it needs information?

You basically build logics to have the model start decomposing the prompt into sets of things it would potentially need and then go fetch them.

And it's able to introspect? Like, this stuff I know about, this stuff I don't know about, this stuff I need to go look up. That's kind of cool.

It is not precisely able to introspect, but for example, if you're trying to make a model discuss, say, GCP, you might have a first step be, Or of your LangChain reasoning framework be, what services is this user question discussing? And then you could say, okay, model extracts the discussed service from the prompt. Great, let me go make those docs available to the model.

This is fascinating. The idea of these models reaching out to other surfaces makes me wonder about complicated information leakage risks.

Yes.

But very fun leaks, not just like, I put it there and it leaked, but like really—

Complicated leaks.

Mind-bending, twisting, complicated leaks. Yes.

Yeah. So I think, well, first of all, I would say there's all sorts of potential leaks associated with a casually implemented retrieval-augmented generation scenario. I would also argue that for all that there's all sorts of leaks associated with that, this is still a good way to control model risk because if you go with the per-prompt retrieval option, You can control which data is used by which users, and you have the option to have data available to some users without having it be regurgitated to other users. Now, there's an asterisk there around making sure that you don't automatically feed prompts back into model tuning. I think for many use cases, it is probably a best practice not to automatically feed prompts back into model tuning. But as long as you do that, you know, you control what is going into the model weights, and therefore you can make sure that bad things don't come out. Now, to your point about data leakage, the thing I would basically note is that when you start using some flavor of retrieval-augmented generation, you're effectively increasing the surface of what acts like code, insofar as the code is the stuff that is controlling your result. So, you need to, the same way that you should have your code under source control, and you should think about who is allowed to access it, and how it goes into production via your build pipeline, You should be thinking about what is in your retrieval-augmented generation corpus, who is allowed to access it, how is it modified, how are those modifications tested, and what is the process by which they roll out to production. These are all questions where we have decades of experience controlling that for code. The change is just that we need to start thinking about the surface area of what is acting like code a little bit differently.

Okay, so this is super fascinating. Listeners, I'm making the face that I often made when I was working for Katherine, which is just like, utter bewilderment as I tried to catch up with what she was telling me. I want to shift gears a little bit and not to name names or not to pick on somebody, but Anton and I had a conversation with a security vendor who were very excited about their implementation of LLM use with an LLM vendor. And they took pains to tell us, don't worry, we remove your IP address before we send your vulnerability data over for them to— give responses about. But this just blew my mind that they were giving my vulnerability data to a third party and they thought the IP address was what I would worry about. How do we help security leaders reason about all this? What's the right vocabulary? And how do we not build things that are that scary and then cavalierly be like, oh, but we made it safe with this one simple trick?

Right. And ideally, we should not have to say the word regurgitate too many times because I just can't pronounce it right. I will hardly try. But I get the concept, but I think that there are other risks of leakage apart from the re-g-g, that thing.

That one. You could call it vomiting, barfing it back up.

It's actually not quite— okay, fine. Subtle details.

I am going to leave the verb selection to you two. But I think a lot of this is, again, things you can frame in relatively classic terms. So imagine that you were doing business process outsourcing 10 years ago.

Hmm.

You know, you would not just care that your IP was being stripped from the dataset to these random humans. You would care that the random humans were properly vetted, that the appropriate privacy policies were in place to protect your data. And you would care that perhaps for some classes of truly sensitive data, that it was not given to random humans who you had no visibility into securing. I think there are similarities here. You are handing a bunch of data to an outside processor. Think about the privacy policy, think about the terms. And this doesn't need to be a specific LLM thing. Like, the terms should offer you a level of protection that is appropriate for the data and for its sensitivity, or lack thereof, to your company. And it should be on your partner and on your contract language to ensure that those terms are appropriately enforced. I think At that level, it only kind of matters that it's even an LLM inside. The thing I would note is that there may be vendors who say, oh, we need prompt data for this or that. I would push back very hard on that. Vendors should be able to offer you services without needing to examine the specifics of your prompt data. It should be possible for them to offer you generative AI under terms that respect your prompt data the same any other service would respect customer data.

Okay. I like that answer a lot on the last part. I guess what scares me and what might scare some CISOs out there is the idea of relying on terms of service as a data protection measure.

Yeah. I mean, but would you rather prefer a rule-based DLP with rules written in the 1990s?

Oh, I'm not suggesting that.

Yeah. Sometimes the tech safeguard is equally naive. And I mean, maybe that's where we want to talk outputs, right? Prompts and Making models spit out data. Spit out, that works.

Spit out, that's nice. You can say that.

The outputs make me nervous because, again, as I was trying to explain data security implications of some of this, I kind of said, well, you do DLP on the input for your normal systems because you don't want bad data coming in. But ultimately, no surprising data would come out of a traditional system. LLM-based system may produce something that would surprise you on the output. So how do you keep the output of a model safe? And how are we And by the way, safe is also a bit of a loaded word. Like, is this really a security challenge? Yes, no, maybe. So in light of this, how do we talk to vendors of models? How do we prevent or detect that models are talking nonsense or doing something harmful with an output or possibly doing some other bad things in the output? So what are we doing with that? How do we handle this?

So I think this is actually a very interesting And meaty question. I'm not sure there's a single right answer. I think depending on the use case, in some cases it may be relatively simple to make sure the model is doing what you want. The way I think about it is that there is a spectrum of use cases. On one side of the spectrum, you really want the model to kind of stick to an established script with small semantic variations. An example of this would potentially be if a model is doing tier 1 support roles. In that case, when you think about the classic job, you really want people to adhere to a very well-documented prescriptive path. And if things start to go off the path, it's no longer tier 1 support. You need to escalate or kick things over to another process. In those cases, there's techniques that you can use to say, the response must be in line with this document corpus. The question must be something that aligns to these sets of known questions. And You can start to ensure that it's within a very walled set of use cases. There also may be UX techniques you can take in those cases. Like, for example, you can avoid potentially letting users ask a completely arbitrary question. You can use UX techniques to force users towards specific sets of questions where you've vetted the data, you've vetted the process.

Mm-hmm.

On the other end of the spectrum are tasks that, where you really want the model to be producing novel content of one form or another. And I think the more you have novelty, the harder it is to completely control the process. The need for novelty manifests as one of two things. One is extreme creativity, and you see this with generative AI imagery, where people are often asking the generative AI for images that definitionally do not exist in existing stock art corpuses. The other is actually insights. I would argue that an insight is boils down to the creation of some statement or assertion that previously did not exist, at least not in the context in which you're operating.

Okay, I buy that.

What is a hallucination except an insight that was actually bad? I think when people do this, you say, oh, that was a bad idea. When a model does it, you say, oh, that was a hallucination. It wasn't true. Tim, I've talked to you about this a little bit, but as a serious people example, I think Most people who've talked to 6 or 7-year-olds will see that this is actually something humans take a lot of calibration on. If you talk to a 6-year-old, they will tell you about the conversation they had with the Easter Bunny.

Sure.

And they will be enthusiastic and detailed. And it is a complete— I don't know if you want to say hallucination, but it is an example of a system establishing the norms around creativity and the context in which it is warranted.

And actually, funny enough, at the recent Gartner Summit, this is exactly the model the analyst presented. He said, stop thinking about this in these crazy ways, think about it as talking to a 5-year-old. And that's why this specific analyst thought that prompt injection is kind of a bad model for thinking about it because it's almost like, do you know of 500 tricks how to deceive a 5-year-old if you're an adult? Sure, you can come up with that. But is this a big achievement?

No.

So that was the same take, that the prompt injection ultimately— the damage from prompt injection has to be handled somewhere else, not by prompt filtering.

Well, my take is that for creative use cases, if you are asking the model to be creative, then you are fundamentally asking it to do things that will be a little bit hallucination vulnerable. And so in that world, you need to have other mitigations, be it UX, be it the way that the model's results are deployed, where you think that the model producing inaccurate data is not going to be harmful.

I've never thought about prompt injection as getting into the truck that says candy, but this is a strange metaphor. Does this mean that hallucinations are just inherent here and if you want it to be creative, you have to accept that it's being creative? Is that what we got to here?

I would say to some extent, yes. I think hallucination has a negative connotation and creativity has a positive one. But they really both refer to the act of creating information or something that was not a previous strong assertion. Huh.

Anton, next time you come up with a weird metaphor, I'm going to call it a hallucination.

That's probably— I sometimes say that myself. I say, am I hallucinating or blah? That's actually the language I sometimes use. So in that sense, I think that demanding that models be just superior to humans in this regard is silly. So maybe I think that the whole fight in the windmills/hallucinations is also a misplaced battle. This is really interesting. So let's go to more specifics, hopefully more useful. This is very fun, but let's try to be also useful. So we are using some of the stuff for security. We are not just securing this, we're not just studying this. So it sounds like there are risks, there are opportunities, but how do we know what are the good opportunities? And I think in the past when we had guests talking about AI, We brought up the same example. There was this vendor who wanted to use LLM, actually just stock consumer LLM with no tuning, to create remediation instructions, basically write Terraform code that people would then run on their prod. And we thought that's beyond belief crazy. But are there great uses/bad uses given what we just discussed?

Yeah, I at least think there's rules I use for thinking about uses that are better versus worse. And I think there is potential for great uses. I'll start. I think great is easy. I think if you have a use case that actually leans into the creativity, then you're taking advantage of not just the LLM's language capabilities, but you're kind of turning the thing that is classically discussed in this negative hallucination context into the power of creativity. An example there is, there's been a bunch of discussion about LLMs and fuzzing.

Hmm.

I think that is potentially a great use case because fuzzing really leans in on the creativity, and there's probably no such thing as a bad hallucination in the context of fuzzing, right? Yep.

That's interesting.

So when you think about things like that, I think that starts to potentially check the boxes for great. For things that are good, I think a lot of good use cases will need some combination of language skills, things that benefit from a semantic understanding of underlying data, and things that benefit potentially from merging 2 or more disparate data sources. So if you need to combine a user question with some amount of contextual documentation, with some amount of other English language corpus, That's something that's really hard to do with classical methods, but where LLMs might shine. Another property of a good use case is ones where people can recognise a good answer. This kind of goes to, if you have a use case where you're worried about something like incorrect results, the more a human can be kept in the loop and can tell if the answer is good, the more you have a mitigation available. And then riffing on that, the more a human can not only recognise a good answer, but the more a human can take an answer that's like 95% good and easily adjust it to be 99% or 100%.

Yeah.

good, the more you can have a clear path to launch because you can start with the thing that's pretty close and then you can tune over time. I think code generation use cases often look like that. Users, particularly given an IDE and given test suites, can tell if something is pretty close and then they can iterate and get you all the way there. Those are things to consider for checks on positive boxes. For negative boxes, I mean, first of all, I do think there's a little bit of LLMs are the new hotness, so people are trying to reinvent the wheel. If you have some AI system and it is working great and you are happy with the results and it is not generative AI, it ain't broke, don't fix it, man.

Love it.

A bad use case is trying to reinvent something that's already working great because you think LLMs are shiny.

So actually, before Tim goes to the closing question, I had an epiphany based on what you just said.

An epiphany or a hallucination?

We'll know in a month. We'll find out. We'll find out. The harm/risk/damage happens when you look at a creative use of LLMs, but you apply it to an area where you need rigor or where you have a rigorous use case. So basically, confusing rigor and creativity is what causes stress. If I want to write creative copy, creative and hallucinatory stuff is okay. If I want to try to write audit report that I would then sign in blood or my CEO would sign in blood, and I'm going to use LLM, then I am—

Bad choice.

—doing it wrong. So this rigor versus creativity balancing act is what we need to have in mind all the time.

Yeah, and so one of the other things I would cite as a red flag for this may not be the best use of LLMs is if the answer absolutely has to be correct with no excuses, no exceptions. So I would not be excited to have an LLM do my GDPR review. That needs to be right. A human might not be the best at reviewing a very large review in that scope. So, you could imagine there are steps of that review where an LLM could assist a person, but I would not be keen to completely hand over the reins. And then, another example of something that I would be cautious about is there are some tasks that are very, very hard to break down into individual steps. So, a security example of this is saying, write me a detection that would catch any way in which Cozy Bear might be running around my system. Ah.

That's not going to work.

Yeah, it's not going to work. This is an extreme example, but the reason it's not going to work is that I could ask 100 expert security humans to do this task and they could each enumerate a pile of things and their enumeration would fall short. This is an example of just because I can express the desire for this task in English, it may or may not be tractable under any circumstance. The fact that the LLM can interact with a request in English is not going to change the tractability of the actual request.

All right, I hate to do this, but we are at the point where I have to ask our traditional closing questions. One, do you have a tip to help people improve their outcomes using LLMs? And then 2, very broad aperture on this one, do you have recommended reading?

So tip for improving things with LLMs, I would say look for places where language skills are useful and you can assist a human while keeping them in the loop and hill climbing towards an excellent option. I think those are kind of good places to get started, to learn how the technology works, and to be able to Do something bite-sized in a realistic timeframe. Recommended reading. I actually think this one depends exactly on what you're trying to do. The more you're thinking about using LLMs to help generate specific programming language, go read a bunch of research papers, the open-source projects. The great thing about this space is that because it is so active, you will find people publishing about whatever specific thing you're looking for. Download the PDFs and get going.

Awesome. Katherine, I can't thank you enough for joining us today. I think all of our listeners would like to thank you for saying, yeah, sure, why not, 3 and a half years ago now. So Kathryn, thank you for joining us.

Anytime.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikin and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode. Security.