EP166 Workload Identity, Zero Trust and SPIFFE (Also Turtles!)

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior PM for Cloud Threat Detection, Investigation, and Response in Google SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with your hosts and the rest of the Cloud Security Podcast listeners on our community page. You can find the link to that as well as our LinkedIn page in the show notes. Anton, this is a special episode for me because it features one of my SHAPE colleagues, which I, you know, represent the SHAPE mafia strongly in life. It's also, I think, a fun episode because it brings together so many of the things we love talking about on this show. What did you think?

Initially, I started thinking that this would be one of those IAM episodes we sort of, like, quote unquote, have to do because IAM matters and it's workload identities and it's kind of interesting, but maybe not really. But ultimately, it was a really good episode about the turtles.

The turtles.

I think the turtles is what did it for me because—

Yeah.

Sometimes if you solve shallow, immediate problems, you solve shallow and immediate problems, but you never solve the underlying problems. And I think the crew on the show is solving some deeper underlying problems. Of course, they may succeed, they may fail, who knows? But they are doing that, and that matters.

I thought one of your questions was really interesting that I didn't expect you to ask maybe so directly or spicily about whether they were the right people to be solving this problem. But I thought their answer was quite compelling.

It was really compelling. I feel like they have thought about this, and this is very tricky for a smaller company to solve a fundamental bottom turtle kind of problem, and they are very well equipped to actually solve that problem. So to me, that impressed me quite a bit.

Yeah, it was surprising, I think, to both of us. And so maybe without any further ado, listeners, let's turn things over to today's guests. I'm delighted to introduce today's guests. Today we are joined by Evan Gilman, co-founder and CEO of Spiral, and Eli Nesterov, co-founder and CTO of Spiral and an old Shape colleague of mine. So it's a delight to have a Shaper on the show. Guys, I'm excited about this episode because it's in so many ways a culmination of some of our favorite themes on the show. We're talking about identity access management, we're talking about zero trust, we're talking about security made easy. Evan, with that kind of glowing intro, could you give us the 30-second version— and it doesn't have to be 30 seconds— of what workload identity is, why people need them, and what's so spiffy about spiffy anyway.

Should we have a detection angle too? Because this is like the only remaining topic. Yeah, good. So if you have a threat detection angle, that would cover literally all of them.

We'll do that later. We'll do that later.

Kevin, what do we got?

We can always add it.

You know, I've, having started a startup recently around this, been thinking a lot about, you know, how you succinctly kind of message and describe this, what would otherwise be like fairly abstract idea for most people. After a lot of kind of A/B testing and all that stuff, you know, the one thing that I found that resonates with most folks and kind of quickly gets the message across is comparing this to, you know, the evolutions that we've seen in, like, human-to-software, like, human authentication and identity security over the last 10 to 20 years. We understand, pretty much everyone understands, my parents understand that You know, when you access a digital system, you need to kind of authenticate yourself so that system knows who you are. You know, over the years, we've gone through these evolutions of like, okay, you know, we have this password manager that we interact with, and we create these accounts, we do all this stuff, you know. And over time, we've gotten better and better about that. And now, state of the art is things like passkeys, SSO. You know, there's a shiny button that you click that says sign in with Google, And no matter where you go, what websites you visit, there's some notion of, hey, this is Evan. Like, we know who Evan is. And there's like an establishment of trust around this. And most importantly, reduces a lot of friction on the user and also enables folks to kind of like correlate these things across properties and have a single place where this is managed and all sorts of advantages there, right? And so when I think about Spiffy and how to like succinctly say like, hey, what is this workload identity thing? Like, why should I, even be worrying about this? Well, it's the same reason that you worry about human identities. The same way that humans access systems, systems access systems. And especially over the last 5 to 10 years with microservice explosion, all this kind of stuff, this happens more and more frequently. And so, this is a very common area of concern when you think about lateral movement and other things in kind of the common attack chain. And so, it's just as important to identify and secure workload-to-workload interaction as it is to identify humans and secure human-to-workload interaction.

When I hear this, by the way, sorry for interrupting, When I hear this, I'm always scared by the specter of rulemaking. Yes, you have identities, but then you have to decide whether at 5:00 AM on a workday, X can do Y. Like, is there this authorization specter of authorization burden? Oh my God, that's— I just made it up. Is there this kind of specter in here? And if yes, then how do you deal with that?

You know, Evan, you should be really happy Anton asked that question about 5:00 AM. Usually it's a 3:00 AM test, so he let you sleep in 2 hours on that one.

Yeah, that's a different model here, yes.

All the same questions apply. The human analog applies in many ways. So, you know, hey, should workload A even be talking to workload B? Does the information that workload A is requesting make sense or not for workload A's set of responsibilities? So absolutely, we call that the authorization problem. In the human realm, authentication and authorization are often very tightly coupled, but in the workload realm or service-to-service communication, we often see a much looser coupling between that. We see identity coming from a number of places. being rooted in a number of things. And then we see authorization being a function of that, but largely orthogonal. And we can see, if you look at things like CNCF Landscape, you can see this happening where there's, hey, these are the identity providers, authentication things. Here are the authorization things like OPA and Cedar and all these others.

And authorization scope is, I feel, as complicated or even more complicated as identity or authentication, right? And like, To use this kind of human analogy, when you use systems like, I don't know, Okta as example, right? So you specify users and group and what applications they do have access to. Like I have an access to Workday, right? And this is what you, what you can define in Okta, but you do not specify in Okta what permissions I do have.

Mm-hmm.

In Workday. This is like very application specific and authorization is like very, Application-specific in this sense. And we see the same kind of in workload identities, microservices, cloud-native world space as well.

I'm hearing it's a complicated problem, but your solution, your approach doesn't make it worse, but ultimately it remains to be solved elsewhere, right? Like, this is maybe— I'm cherry-picking from your answer, but this magically solves problems, just not this problem, right?

One way to think about it is like passports. So, the U.S. government, we're in the United States here, I think most of us maybe.

We are for the recording.

Our listeners are around the world. Your government will issue you a passport and there is a notion of governmental identity behind that passport. They're not necessarily responsible for issuing you the visas that you need to travel to the countries you wish to travel to.

That is Oh, this is a cute analogy. I like that. I think that's super reasonable.

Okay. That makes sense. But then wouldn't it be— then it's also the other analogy or the other connection I have is kind of to the microsegmentation when people try to achieve, vaguely speaking, the same goal by configuring which workload can talk to which another workload, which system can talk to another system. So, is there kind of pros and cons or differences between this approach that you're Highlighting and micro-segmentation.

You want to take this one, Eli?

Yeah, totally. The micro-segmentation is just like what it says, right? So we used to have this like large parameters. Now shrink them down to like, I don't know, hosts and workload. But the biggest change right now for the last, I don't know, 5 years is we have a lot of workloads and they're very dynamic. So making this micro-segmentation rules working is like you basically go into this rules burden, right? But it's like now, now you're dealing with this like IP addresses on a different level. It's like hundreds or thousands and they're pretty dynamic.

Mm-hmm.

Mm-hmm. Is identity-based, I don't know, authentication is why it's better here. It's like, because you don't care about where it's running on a host A, B, or like in which region. In this case, so you operate on a high-level perspective. Can service A talk to B or C at 5:00 AM, right? Because there is like attribute-based access control on top that you can, you can put, of course. And it's way, way easy to manage.

That makes a lot of sense to me.

Does it mean you shouldn't use like segmentation? Probably not. It's kind of a defense in depth. But it should be a reasonable amount, right? So, you probably want strong isolation between your dev and production environment, but within production environment, you also want to have some specific rules.

That makes a lot of sense to me. I think I want to switch gears a little bit. I was doing the research for this episode, and I saw that you wrote a book, and it had a name that I just loved. Could you explain what the heck you mean by solving the bottom turtle? What does that mean?

That one was a little bit of a play on our side and just kind of—

A little bit?

Yeah, in the workload identity circles, we tend to joke about this. And, you know, there's an old story about somebody who said, oh, you know, the world is held up on the back of a turtle shell, and like, that's the story of the universe. And then somebody was clever enough to say, well, what is that turtle standing on? And then the response was, well, another turtle, of course.

Turtles all the way down.

Exactly. So, that comes turtles all the way down, and just be careful when you pick up that turtle. they may find another one. And this is like the story of infinite regress, and it comes up a lot in security.

Mm-hmm.

You talk about, hey, I have, you know, a database or some encrypted file, and I need a key to access it, but how do I get that key? Well, you need another key or thing to access that key, and then how do you get that key? And so this turtles all the way down story resonates a lot in this kind of like secret management access workload identity world. When we think about workload identity in Spiffy, we look at it as, Hey, this is the bottom turtle. This is a turtle that it stops here. We have a way to issue you that credential from zero, to go from zero to one. That's the hardest part. Once you have that one, all the rest can be solved through exchange and other things like this. And so it's kind of a play on this idea of turtles all the way down.

It's depressing that it actually makes sense to me. This is actually kind of sad, not because I like the analogy, but because people often refer to Oh, yeah, this is a hard security problem. We'll just rely on this other system. And of course, in most realistic cases, they say, we're going to rely on Active Directory. And then everybody just cringes and runs away screaming, right? And then people say, yeah, that's a great idea. This pays our bills.

Oh, that's depressing.

Okay, fine. At least I'm humorously—

Yeah.

So, SPIFFE is a framework and basically a specification that describes, like, how do you solve it. How do we solve that problem? And there is an open-source implementation that's called Spire that basically implements the specification. So you can use it as a building block to build something internally and run on your own.

Nice.

But these problems are also hard to scale, right? These are usually— it's even when you solve for the proverbial bottom turtle, often what you do is you solve it, but it works for the stack of 3 turtles. But in reality, well, there are many more stacks, not only there are more turtles. So how is the scaling of this? Like, again, I'm back to my ghost of rulemaking on identities and workloads, but like, what is the automation magic? What is the way to actually have it cover a large company with messy IT?

One of the really critical aspects of Spiffy is that it's platform agnostic. And so it doesn't, anchor its identity and credential lifecycle scheme on any one kind of, like, cloud provider, runtime, things like this. And this is what brings a lot of power for these really complex environments where, hey, you know, I've got a little bit of this and a little bit of that, and how am I supposed to wire all these things together? Spiffy acts as kind of the lingua franca. And so, it's actually at scale where the value really shines at this scale. But I think that, you know, the better person to kind of talk to this point is Eli, since Eli ran, like, one of the world's largest Spiffy deployments and has kind of that firsthand battle-tested knowledge on what it takes to kind of roll this out on massive scale. And so I defer to him on, on most of this stuff.

Thank you for putting me on the spot. Yeah, so it's like running an open source SPIRE is like pretty big lift because first is security system, right? That's dealing with the key and key management, and it's a PKI itself, and it's also issuing your JWT type of identities. So yeah, it's like dealing with keys and everyone kind of knows how to deal with this, but at the same time, nobody knows it. And it also has a database component. So it's like when you're talking about scaling, you need to know how to scale the database eventually. This is a critical component that everything in your production environment depends on. So it's like, there's like different levels of complexity. It's easier to start with kind of one data center, one region where you have like a small deployment, but Mm-hmm. As soon as you're talking about multi-regional, multi-cloud deployment, complexity of that system has started to grow. So yeah, that's what's like, we took that experience and basically rethink there how you can run the things and what is the hardest part about like running open source when we started the company. The way we think about it is like, hey, here's a, I don't know, here's the Kafka, you can run it on your own. It also like takes some time and a team to scale it up and scale it down and run reliably. And there is Confluent as a solution. So basically when we, when we think about Spiral, it's in, in the same terms. If you want to have workload identity, SPIFFE is a standard.

Mm-hmm.

There is different ways you can get it. You can get running open source, you can work with Spiral, you can get it running too. You can get it running Anthos, you can get it running some Azure applications as well. So it's kind of started to pick up in that sense.

Yeah. So I don't want to play devil's advocate here too hard because I like both of you guys. But if I'm putting myself in the shoes of a CISO, I've got a lot of stuff that I might imagine doing before this. And now I know from where I sit that this is workload identity core to how Google does security. When I read red team reports, often people are frustrated by exactly these controls. But how do you get somebody to prioritize this? How do you say to a CISO, this is the thing you got to work on this year?

And it's in a stack of other things. It's not just like point at the thing and say, this is the thing. It's ranked somewhere in a stack of things. And if your first stack of things is, maybe we should update Windows this year because we did it last year and didn't like the experience, like for that company, probably, Not the high priority, but others.

Yeah.

Yeah.

No, it's a great question. I think if you look into CISO's top list, there won't be like workload identity. I think it's like you have a good word there as tech. It is in the stack and the stack would fall into category like an access control or a better authorization, right? And it's always like companies and people always trying to minimize the risk and solve enough authorization and better access control. But when you have this kind of balkanized source of identity inside your production systems, right? These services use API tokens, these services use like Kerberos, something else. It's very, very hard to solve an authorization and access control without having this kind of single source of identity. And SPIFFE solved this problem very well, being this kind of layer of unified identity platform. I personally dealt with this at ByteDance TikTok when we need to build like better authorization access control, but we realized we have like 5, 6, 10 different identity systems. And if you build in authorization on top of it, that would be like a nightmare.

Hmm.

Yep. You just basically have to go down to stack and solve this identity problem that makes like an— your authorization and access control rules way, way easier. But it's not like it won't be as a workload identity as an item on a CISO list. It will be under this kind of access control list for sure.

Make access control actually work, make access control palatable, resolve, banish the ghost of too many rulemaking by too many manual means. I think I get that.

It also depends where on a stage on a company. It could be like, hey, we need an Encryption in transit everywhere, or we need an mTLS, right? And because like SPIFFE produces identity in a form, one of the forms is X.509, you can use it for mTLS as well. So it won't be like workload identity, it would be like mTLS in this case, for instance.

By the way, another question from Tim that I've learned is not only what gets better when you deploy it, but also what should go away once you deploy it? Is there an old approach that you suggest people toss in the trash and use yours? Is there any kind of replacement argument?

Yeah. Is there a peace dividend to deploying this thing?

Tim didn't actually say that, so I've just made it up.

No, I like this question. I'll take credit for it. What do you get to throw out when you buy this thing?

When I think about that, the real big thing I think about is I made comment to that user identity journey before. What did we throw out there? Well, we threw out all these click ops around LastPass and 1Password. And I think the same in, you know, the service side in the data center, you know, what are you throwing out? Hey, it's like, what do we need passwords and API keys for and all this other stuff when we've got like a first-class notion of identity that's interoperable and standardized? You know, so when I think about, okay, let's take this and draw it out to the infinite scale, get rid of, you know, this kind of secret ops and people being called to unlock vaults and all this other kind of manual toil, that's what goes away.

Which is very fragile, resistant, not resistant to, you know, vulnerable to a bus test.

Yeah, you know, vulnerable to a typo. I mean, that's, that's a big piece of resiliency you pick up there. I like that.

Yep.

Yeah, another thing is like this kind of heavy end-to-end automation, right? So it's, uh, if you need your services to have TLS, you need to provide them certificates and the keys. They usually have like some TTL, right? So we started with like a year or 3 years ago wildcard certificates. Now we are going down to like a 90 days, 60 days, but SPIFFE can take it to like off to up to few hours. You don't need to kind of worry about, you kind of remove a human in a loop that needs to like go and apply to some internal PKI, get the new thing signed and put it somewhere in a secret manager. So this is another thing that it solves, like kind of removing these manual processes. And by lowering down, like, I don't know, imagine you're going from 90 days even to 24 hours, like your certificates, that's an automation. You worry less about some of them being compromised or stolen. Right.

That makes sense.

There's a lot of upside to that. Yeah. Anton, you want to ask a spicy question?

Yeah, no, it's actually not that spicy because one thing that comes up is that if you put in my former Gartner hat on. If you observe the very tactical problem that people face, you build a startup, people say, we suffer from X, you say, I have anti-X, you fix X, they give you money, everybody's happy. And this is kind of a top turtle problem, to use your logic. Here, you're aiming at something pretty deep in the stack. And is this really for a startup or a new company to solve? Because And I don't want to make any more Active Directory jokes, but some of the pieces of infrastructure are so entrenched that it's kind of hard to replace them. I mean, if you build a better identity store, it's a long journey to get it adopted, and you probably should be a large company to do that. So, is this a startup-type problem? And if yes, why yes? If no, why are you around?

That last part was spicy, you have to admit.

Okay, fine. The last part was spicy.

At the end, you caught me with a different angle. I thought first you were going towards, do startups need this? And then I realized. realize later on.

No, no, it's more about the type of problem that startup can solve realistically.

It's more about should you exist, Evan? Should you exist?

Yeah, I must fight for this now.

Yeah, please.

You know, what I would say to this is, hey, look, you know, if we point at Active Directory, if we point at the things which have historically kind of been entrenched in these areas and solve this, and we say, okay, what does Active Directory do for this problem in the service-to-service world? Not a whole lot is the answer. And what has happened and what we've seen over the years is as people pay more attention to like the security criticality of these cloud workloads, or even inside your own data centers, not to the public cloud per se. But as people pay more attention to this, they start to apply these existing technologies and existing adoption like Active Directory, OAuth, and things like this to this world, and are met with never-ending pains because this is just not what they were designed for.

Right.

These things operate at a whole other scale than the humans are, with headless interact, all this kind of stuff. And so, You end up kind of really back towards, if you, like, if you try to apply OAuth on these problems, what you end up with is client secret and basically back to username and password that you have to, like, configure onto an app somewhere. And so Spiffy has really kind of reimagined these same paradigms we know, like, we know how this show goes, but we've done it in a way that is, like, very highly ergonomic and automated specifically for these service-to-service interactions. And they're kind of different beasts. And so we see these worlds come together on the edges, you know, when human interact with system that then interacts with other system. What does this togetherness look like? So, we see these things coming together, and even that is still kind of evolving to this date with active work in ITF and other areas. What we have now, and kind of like the entrenched options, certainly do not cover in an adequate way operating this stuff at scale, and that's where Spiffy comes.

I think you possibly mostly convinced me that it doesn't require you to be Google or Microsoft or somebody very large, somebody very foundational. I don't know what's the technical term I would use in my analyst years. Like, it can be carved up and shipped.

I'll give you a very good example of exactly this.

Okay.

I was speaking to somebody recently who said, hey, we have Active Directory, and we have all of this stuff, like, baked in Active Directory, Group Policy, blah, blah, blah, all of this thing. Oh, there's, like, this OAuth bridge, and there's a way we do OAuth there. But, you know, we have like these cloud workloads, and they come and they go. And anytime that we want to launch a new cloud workload, we have to go and open a ticket to create a service account in Active Directory.

Oh, no.

And then there's a bunch of material exchanged here, and then a human then goes in and drops this over. And so this automatically, right in your service development cycle, is taking weeks, you know. And so this is like one kind of concrete way, like, where these things fall over, because there is no, like, highly dynamic automated enrollment at scale for Active Directory.

If you leave it to large, mature, 20, 30-year-old enterprise companies to solve it, we'll end up with like a ton of micro-segmentation probably.

Hmm.

So we, like, I am convinced we need fresh thinking in a lot of all these problems. And Spiffy built like on the experience of how Google, how Facebook, how Netflix running this in a cloud-native environment. This is the company who pioneered it, like who've been starting dealing with what we call these days cloud native, like 10, 15 years ago.

Mm-hmm.

So it takes kind of, it's the ideas of Spiffy built on top of all this experience. And now what we do is trying to help enterprise companies who now like understand, okay, well now we need a cloud native. Now we know how to run things on Kubernetes, how to adopt this. security paradigms and deal with them in this new world rather than trying to use, like, okay, how do we use like micro-segmentation with automation with this kind of new and dynamic world, basically?

So listeners, you couldn't see this if you were just listening, but if you tune in on our YouTube channel, you can see the face I made when Evan was describing that process of passing keys around. It's the same face I would make if I saw a car accident, a train wreck, a plane crash. It was bad. And so I think that sort of contrast, Evan, gives me hope, maybe even more hope than Anton expressed in his, you almost convinced me, which is high praise for him, by the way, about the value of the startup you're doing. I hate to do it. I have to ask our traditional closing questions because we're at time. One, aside from solving the bottom turtle and aside from Surely You're Joking, Mr. Feynman, where that story comes from, Do you have recommended reading for our audience? And 2, do you have a tip to help people improve their workload identity security adoption journey?

I was absolutely going to recommend the bottom turtle. I wrote a book some time ago called Zero Trust Networks, which was, you know, the first book published on the concept of zero trust and talked, you know, in part—

O'Reilly, right?

Sorry. Yes, O'Reilly.

I read it. Holy cow.

I didn't realize it was you. Ha, that's so weird.

Yeah, it's that Evan Gilman.

I believe there's a second edition on its way out now. I recommend it in part because it talks a little bit about this, but also because it is much broader in scope and kind of gives the understanding of what is going on in the security industry today. Like, this shift that we're seeing now, workload identity being part of that, what is driving that, where does it come from? I'm ashamed to plug my own book, but I will put it there just because I think it's a good overview. And the second edition, I was less involved in. I might recommend that one. But I would throw that out there. And we also have a Slack organization, a Spiffy Slack organization. You just go to slack.spiffy.io. We've got thousands of people there who hang out, talk about workload identity, happy to answer questions and do meetups and all that fun stuff.

I love that. And Eli, do you have advice on adopting?

I would +1 on that. Yeah, honestly, it was 2020 when we wrote this book, and we, like, I wrote, like, partially this for myself as well. There are like lots of wisdom and answers on like why, why you should be doing, how you should be doing. And, uh, we wrote for lots of people who is trying to adopt workload identity, how to reason about it, how to actually do a sell internally in your organization that you as an organization should adopt it. So there are lots of kind of things like this beyond just normal technical book in terms of how do you reason and roll out in, in production because it's complicated system and it takes group of people to run it. So how you reason about it. And the book that Evan wrote with Doug, Zero Trust Networks, I think this is kind of very, very high, like this is a good start in terms of in general understanding the systems. And I think it'll help in the industry as well because I feel like these days people talk about zero trust and it became like this really commercial word that people usually means like, oh, that means a VPN.

Oh no, that's a depressing take for the very end of the—

Well, maybe on that depressing take, listeners, we'll wrap it up there. Eli, Evan, thank you so much for joining us today. This is a lot of fun talking about Turtles and identity and how we go about helping organizations secure their workloads.

Thank you for having us.

Thank you both.

And now we are at time. Thank you very much for listening and, of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikinen. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.