EP192 Confidential + AI: Can AI Keep a Secret?

Secure running. Hi there.

Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior PM for Risk Analytics and Cloud Threat in Google SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us, and the rest of the Cloud Security Podcast listeners on our Google Cloud Security Community page. Without any further ado, let's turn things over to today's guest.

And today our guest is, again, Nellie Porter, Director of Product Management for Cloud Security at Google Cloud. Welcome to the show for the 3rd time, Nelly.

Thank you so much, Anton, for having me on the show. It's a very special show for me because my podcast was the number one podcast ever recorded for this show. So, it's always really, really great fun and honor to be part of your show.

That is true. You sort of launched us in the stratosphere, but we want to hear a bit about your story. This is the Security Voices collaboration. So, Jai is my co-host today to ask you some fun questions, not only about the technology, it would be quite amazing, but also about your story. So, you're doing confidential AI at Google. How did you end up doing this? What's your story?

If I will step back a little bit, Anton. So, I started in Google 8 years ago. The time is flying very, very quickly. And again, my first few years in Google was fantastic because it was like a kid in candy store. I was able to touch a lot of fun projects that Google was running back then. Back then, we introduced first ever Titan, our ASIC.

Yes.

It became the history and became so important here later when we explained to the entire world that Google invented and designed a new, completely new chip and made, again, integrity of the hosts and their identity, critical part of our security story. I was also part of team that worked on hardening our hypervisor because our KVM is very special KVM version.

Mm-hmm.

That ensures that we will not expose unnecessary 100 or maybe 500 types of mouses and their drivers to our cloud customers. So, hardening and make it very small from trust perspective is a very critical area for me. And from day one, confidential computing was a journey that I started in Google. We had such a difficult journey with confidential computing because the whole entire industry was sold on very complicated, quite secure solution that was completely unusable from cloud customers. And I probably been one of these people that advocated that security cannot be accomplished without usability, performance, and scale. And it's exactly where we've finished up when we announced first solution to the market. Since then, we're rolling confidential computing to multiple areas. And no surprise, with AI and ML and GenAI, there is so much attention on privacy and confidentiality of models and data. So, it's a fantastic combination, this fantastic opportunity to make this technology very useful in this new era of AI. It's about me.

So, it sounds like you're having fun still after a good number of years here, right?

Absolutely. Those 8 years was complete fun and continue to be fun. And more and more challenges and more and more fun areas I again acquired since then. So, again, my responsibility actually increase, and now I'm owning encryption end-to-end.

Mm-hmm.

Because, Anton, when you're talking to customers and starting to talk about confidential environments or confidential AIs, they're not only focusing on this part of the journey, they need to ensure that all of the data, when they bring it to the cloud or store it in the cloud and run workloads, AI-related or not AI-related, in the cloud, all of this protected end-to-end. And it's no surprise encryption in transit, encryption at rest with all products related to that, and confidential computing products becoming part of my portfolio.

Perfect. So, it does sound like it's not just fun, but it's also a larger area of responsibility, at least compared to your previous and pre-previous podcasts. So, judging your— if you try to judge your career in podcast appearances, sounds like you have a lot more responsibility. But I want to switch to the fun topic of confidential compute that we already covered in the past and AI. So I know that during the previous episode, we sort of a little, maybe bothered you a little with questions like, who is the ideal customer? Who is the best customer? So for AI plus confidential compute, what problem does that solve and for what client?

If we're talking about AI workloads, we always need to think about its multi-stage story. And customers coming to, again, GCP for different use cases.

Mm-hmm.

So, irrespectively, if it's chatbots or different AI agents they're creating, or they need to summarize some PDF files, so trying to scrap, whatever needs to be done from their internal search story.

Mm-hmm.

There's huge amount of data involved in all of those stages and preparing the data for their AI jobs is one of the very complicated and very convoluted story that all of our customers have to go. One very interesting things that we need to remember Models cannot forget. I think as soon as they realize that, it's becoming so critically important, the value of the proper data. Because if you're putting garbage in, you're getting garbage out. So, to be able to process this sensitive data and create these perfect datasets for your AI jobs is the first stage and first confusion that our customers have. And because it's never sensitive-only data. It's combination of public data and private data and their enterprise data and the data that they receive from other vendors. This combo is incredibly difficult to figure out how to clean it, how to remove biases, how to do all of those steps and running it in confidential environments. It's actually incredibly powerful tool for this task. Because it's bringing privacy first. It means it doesn't matter what is the sensitivity of the data, because combined data, this pure dataset that they're trying to accomplish, has incredible IP and value for our customers. Step number 1. Now, step number 2, we're talking about those different customers, that have different relationship to the model ownership. So, if we're switching our topic to models for a moment, we have those, again, OR 10, OR 100 customers that building their own models, huge models, mid-range models, and they exposing these models and APIs of model as a service to other customers.

Mm-hmm.

to consume. They care about the data that they use for training those models and privacy and integrity of the model itself. They also want to ensure that consumers of this model only use its APIs that they exposed for them to use.

Mm-hmm.

So, all of that required confidentiality. Integrity and privacy for the models and data. So now we can't separate those 2 things because application model and data coming as a wonderful trifecta that needs to run in confidential environments. And the final step, when we're talking about customers that fine-tuning the models, they're using existing models and putting some custom data to make it more customizable for their chatbots and other things. Those customers too that working and using inference with RAG to make real-time data possible for those models. It's again, huge amount of data and interaction with the model. Those interactions also have to be protected end-to-end. So combination of confidential solutions confidential hardware. And again, data, it's what this space is offering. So long story short, and it's where confidentiality has to be expanded from CPUs to GPUs. Because now we have to use those GPUs to process models, to run inference, to run fine-tuning, to run debugging, all of these in-context learning use cases.

That's very interesting. So, actually, I wanna dig in a little bit deeper on that. So, what do you think are some specific real-world applications where you feel the combination of AI and confidential computing has, you know, the most significant impact, like, in your opinion?

We have, again, I can't talk about specific use cases or specific customers, so I have to be a little abstract on that.

Why not use cases though? I mean, customers I get, but specific use cases sounds like—

Yeah, use cases. But basically what I'm trying to say, not talking to specific customers, it's what I was talking about. And you're absolutely right. So, again, one of the examples that we have, is customers that coming from healthcare space. And one of the use cases, and it's actually publicly documented. So, the customer name is AstraZeneca.

Mm-hmm.

As AstraZeneca coming with very interesting use case when they using software as a device to provide different again, measurements of your health during your remote visits in the remote places. So they found that for them to be able to manage this information and run the specialized device and the model, mini tiny models in confidential environment, is provide ability for them not only to isolate cryptographically this specific, very sensitive, again, execution of this, uh, software, this device, but also prove to their users and their auditors that it's done as they said in confidential environment. The 2 things that confidential environments actually is super cool compared to any privacy-preserving technology before. First, it's incredibly trivial to enable. It's incredibly usable and general. It means you can put a lot of software solutions, models, or anything that you need in these confidential boxes where we will protect cryptographic But the second very important thing, it's what we called attestation. And cryptographic attestation is ability for us to prove to the third party that whatever you're running is actually running confidentially. Those 2 aspects is very critical to our customers in specifically AI world.

Mm-hmm.

Because the trust to cloud and trust to the models and trust to this situation when you need— and it's a very important use case that we see across all AI-specific customers— when you need to merge your data from multiple sources. And usually in our world, nobody trusts each other. We called it multi-party, again, compute use case. Cases where customers from, again, different financial organizations have to merge the data to be able to provide some feedback. Money laundering situation, it's a fantastic case. When you can't actually detect money laundering only because you have data of your fintech company, you have to have, again, all data from multiple companies to compare and contrast the behavior of this individual. We have another interesting use case also from companies that was very public about what they do on confidential environments, and it's Mantec, Google. And it's companies that actually working on detecting double linking, double spam.

Mm-hmm.

When you put, again, your, for example, value under different loans that you cannot. So how to detect that it's happening and you double dipping in those, those, again, equities that you put on this loan. So there is multiple opportunities in those use cases. And again, it's using the standard ML, not necessarily GenAI, but also We started to see that GenAI models are also becoming very interested to utilize this technology.

But if these are environments when they have limited trust to the cloud provider, I mean, sorry for being a bit contrarian here, but wouldn't they be better off using GenAI on-premise? Like, if we are trying to use confidential to shield them from us, Wouldn't on-prem be an option then, right? Why is cloud plus confidential plus AI a better answer than AI on-premise, LLM on-prem? You can run LLMs on-prem, right? They can do open source, open weights LLM and just like dump it on servers.

So, Anton, it's a fantastic question between us. And I would say that customers, it's not that customers don't trust Google. Customers would not bring workloads to Google, to our GCP, if they would not trust us.

Mm-hmm.

Customers probably worry, and properly so, about risk of exposing our infra to our insiders, and they're also worried about exposing the same environments to their own insiders. It's the risk, it's the threat modeling that they have in mind. And as less as they need to trust, as less software and access they have to tolerate or accept risk, is better for us and is better for customers. And it's so interesting that you're putting this because I had this conversation again and again internally and externally.

Mm-hmm.

And the feedback is, if customers would not trust Google as a company. They never would come to Google as a company with their sensitive workload. So it's point number one. Point number 2, AI and ML was running everywhere, but GenAI, all these huge models and utilization of these models that train by, again, only a few companies have enormous resources, have to have supported by, again, hundreds and hundreds of very powerful monsters, H100 GPUs, to be able to get these resources and scale in the level that it's required for those customers to do fine-tuning or training or even inference. I'm not sure that it's even feasible to run these workloads on-premises anymore. And it's why customers have to go to the cloud. But they need to understand the security guarantee and our threat modeling of how we protect the workload on and in GCP. For example, we announced at Next that we're coming with confidential GPUs. We've worked very closely with NVIDIA on their newest huge H100 GPUs to make them natively confidential. It's 3 years plus work. And between us, Anton, the story, I, for this podcast, I will give you the whole story. NVIDIA team came to Crew Client where I'm located and had fantastic lunch. sitting there alone, talking to themselves. So, myself and my engineering, again, director bags and passing by, as usual, we're very, again, welcoming people. We are inviting them, asking them. We realized that they, again, from NVIDIA. So, we asked them, do you know anything about confidential computing? 3 years or so ago, they said, nope, no idea. We said, We talked, we talked more later. They asked us to join their headquarters literally a few days later. And since then, our teams worked together to make this first-ever GPU fully confidential from day one. So we're bringing these things to our customers in GCP because having these machines these confidential CPUs and GPUs available, it's incredibly difficult to accomplish on-prem. So we have opportunity to offer this capability for customers to protect the model and the data, an end-to-end story between CPUs and GPUs, fully confidential.

That's very interesting. So I wanna talk a little bit about the various workloads that exist. Now, there's training workloads, there's data curation, operational workloads, which of these needs to run in confidential AI? And I guess a part 2 to the question is, what are the performance or cost implications of this? Like, you know, are there any risks that come out of running things in confidential AI?

Let's, again, have those few workloads or use cases outlined clearly, and after that, we'll map them to your question, Jim. So, again, Training. Let's start with that. So, I have my model, I'm using a lot of data to train it, and I want to ensure that this model continues to be, again, protected and properly managed as my training jobs continue. So, because the model and risk is more protecting my IP, Sometimes data is lesser of problem if it's fully public data or something. It's not the case anymore, but let's assume the model protection is more critical than data protection for training jobs. The second step is what we call fine-tuning. I'm using this model. I don't touch this model, but I'm using the specific APIs that model owner exposed to me to ensure that if I will feed my own data, I can customize the model layers to become much more specific for my, again, financial, healthcare, or something else.

Mm-hmm.

Is the fine-tuning use case. The third use case is inference with in-context learning, or RAG, we called it, when you're trusting your model to answer properly, The questions that your users might send to you for your chatbots or your summarization task, etc. But in particular things, those models missing the real-time information. For example, in Gemini Cloud Assist, something that Google offered and announced next, the knowledge of our recent confidential GPU is not yet in Gemini.

Mm-hmm.

So, how do we ensure that real-time information would be available for Gemini when customers ask, how to protect my model and data in GCP? So, in this model, we use what we call a RAG to do in-context learning and ingest some additional information from our public documentation or from to address customers' needs, to help them with proper deployment, proper solution, proper way to deploy their workloads to this. So it's what we call in-context learning or RAG-based inference. And the fourth area is basic inference. You, again, the users utilizing the chatbots and all other models to send your queries and receive So in every single step on the way, when we're talking about training, fine-tuning, plugging, or in-context learning, there's a different begins right model exists and different risks involved. As I said, training, I'm mostly worried about IP of the model because I'm spending millions of dollars to train this model. In my fine-tuning, I'm worried about both. confidentiality of data, because to customize this model, I need to use very specialized data. Specifically, again, some of that may be public as it happened with us, but for, again, when company is building this customized model for their internal search, there's a lot of very sensitive data has to be done. And as I said, Models can't forget. So now you need to ensure that model and data is protected. The third, RAG situation or in-context learning. In this case, RAG data is important. They have to be protected because it's what real-time information you are feeding into the model together with the context. Again, depending on your history and your window, context window, maybe the sensitivity or necessity to keep it memorized by model is not as high depending on how you configure, what your application is doing. But in multiple cases, we've seen customers want to have what we call multi-turn communication. And it means model have to remember the next words, the next things, and models can't forget. So now the model and the data has to be protected. And the last case, when I'm sending my query and asking model to answer my very sensitive question, those queries and responses is important. So if we can run model and data in confidential environments, we would be able to provide solutions for all of those different use cases. But then, as you said properly, what is the price tag? Nothing is free.

Hmm.

So the price tag from performance and latency, and I think again, definitely additional work is required to do additional encryption and decryption of traffic. Because communication between CPU and GPUs have to be encrypted on the wire on this PCI Express bus.

Yeah.

So, additional work needs to be done on encryption and decryption, and additional HBM, your high-band memory, needs to be used to be able to accomplish that. So, it's some, again, performance impact.

Performance impact really means somebody spending money on this. Like, I mean, I think that this could kind of like somebody, somebody's wallet is leaking dollars or euros or whatever other currency, right?

Absolutely. Performance is always equal cost. Yeah. It's total TCO. It's never only, again, I'm slightly delaying my response to you in your channel. It's money because you need more hardware, more resources to be able to process the same thing. So again, When we talk to customers, confidentiality and privacy guarantees that we provide together with attestation, the feedback that we received, again, under 10%, we will be okay. Over 10% is more than we would be able to support.

It would be a debate, right? And if some VM or some container or whatever stack processing this would cost instead of X, X plus 10%, I will stop her, yes. We have a deal or we have a discussion, but if it's, I don't know, double, then there is no deal. We'd rather figure it out on-prem or—

Absolutely.

Not solve the problem or basically take some other path to success rather than the confidential stack. Well, that makes sense. And I think to me, you know, I don't want to hold you to this, but I vaguely recall in the very early days of confidential compute that even 25% was not uncommon. So, does it mean that we improved the tech? Or something improved, something became better here, right?

Everything is improving.

Life is getting better, actually.

Sometimes it's what we do for a living. But sometimes we compare and contrast orange to apple. And what I mean by that, we know that in confidential environments, CPU only, CPU GPU, doesn't matter. We have to spend time on encryption decryption.

Mm-hmm. Yeah.

So, we try to mitigate it utilizing some really crazy technique or hardware offload of some capability. It's what GPUs do. They actually don't do encryption decryption in software. They have dedicated ASICs that does encryption decryption for GPUs to ensure that memory is encrypted. And they would be able to So, it's utilization of hardware capabilities that are available to us. But also on VMs, as you said, we do a lot of tricks, honestly. We have very sophisticated, for example, we stopped using SCSI drivers for access to the storage. We're using NVMe drivers. Why is it so critical? Because SCSI enables one channel. Literally, you're writing this traffic and whatever you need, your IOPS, through one channel. And vME opening 64,000 channels. So, it's why I'm saying when we have VMs with SCSI driver and in vME, it's the details that, again, my customers are probably ignoring sometimes. But I've seen customers coming to me and saying, Nali, we've done all our perf validation of your confidential VMs, all the confidential VMs faster end-to-end. What is that? It's impossible because you need to do encryption decryption. It's like, okay. We have a lot of different things that we are trying to bring to those confidential environments to mitigate some of expected performance and latency degradation. And this is the way how we work. And this is why, as you said, As time progresses, we're becoming better and better at that.

I think that makes sense. And I think, unfortunately, as usually Tim says, but we are kind of running Timless today, it's time to get to our closing questions. And of course, it's the usual, give us one tip on using confidential AI and give us one resource, one reading material or something of that sort.

And one tip that I will leave you with, and it's so funny, I will be super consistent on my First and my second and third podcast. Running confidential AI is exactly as running non-confidential AI. You don't need to change anything. You will take your pods with your wonderful jobs. You will take your model. You would take your VLLM and whatever you do for living.

Mm-hmm.

And you will get one checkbox saying, run it confidential. And again, with a little bit more performance degradation, yes, we are still taking a little bit toll on you, you would be able to run confidential. So everything that we do have to be incredibly usable and no requirement for you to adapt your application model or anything else to support confidentiality. It's done on a lower level with hardware, work with vendors and everybody. So you, my customer, doesn't need to do anything. Checkbox, pay a little more, and you're good.

Hmm, I like that. And any reading materials, any pointers, any 70-page white papers?

We have plenty of white papers and everything, but Mostly explaining why you will need and what is the value you will provide. It's not what you need to do in your workloads to make and take advantage of that.

That makes sense. Thank you for your excellent 3rd appearance in our show. So looking forward to hopefully the 4th in a couple of years. And again, thank you for being here.

Thank you so much and hope it's not last one. Thank you for having me here. Really appreciate it. Great conversation, Anton and Jay. Thank you.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton underscore Chuvakin and underscore Tim Pico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode. Security.