EP289 Software Engineering vs. Software Craft: How Google Scalably Eliminates Classes of Vulnerabilities

Security. Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, myself, Tim Peacock, Group PM on Google SecOps, and of course, Anton Chuvakin, a reformed analyst and now senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get our podcasts, as well as at our YouTube channel. youtube.com/@cloudsecpodcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. Follow the show, argue with us and the rest of our Cloud Security Podcast listeners on our LinkedIn page. Anton, what are we talking about today other than, of course, how cute Waffles is?

Yes, Waffles is kind of mind-blowing. And I was just seeing his ears peek out from the— into the frame. And I feel like that sets the— that will set the show on a really right tone. Yes.

Yeah, right? Yeah. Everybody loves waffles. Listeners, if you like waffles, drop us a chat that you love waffles.

Yeah, that's gonna be useful. And I feel like we'll know which ones of you listening for the dog and which ones are for Tim, and maybe there are some for the reformed analyst. I don't know.

Maybe some.

We can debate that one.

So today's guest though, they were talking about the elimination of cross-site scripting vulnerabilities, and it gave me a flashback to my junior year at Brown when I was in a security class and we were learning about different classes of vulnerabilities over the years. And we had to work so hard to find a flavor of Linux which actually still allowed you to commit a buffer overflow. And the idea that just 15 years later, we'd be getting to the point where that was the case for browsers and cross-site scripting is kind of mind-blowing that we've come so far so quickly.

I mean, I feel like to me, I had a different flashback. I had a different flashback to when— I mean, the phrase eliminate whole class of vulnerabilities has been copied from Google document to Google document probably for way longer than I was here. Like, I don't know about 10 years, but I won't be shocked if it's more than 10 years. A long time. And we simply treat it as math. There's a set of classes, certain classes got eliminated, we don't have them. But when we present or mention it to the outside, people say, oh yeah, your marketing probably wrote it. And we're like, what do you mean?

No.

Like, that's kind of a fact. There's a, like, and they basically go through this list and people just look at us really funny. And then now when Heather says, hey, actually, with the help of AI, we can eliminate a lot more classes and probably a dramatic set of all classes of vulnerabilities will just be gone. People just, like, get really upset and think that we are making stuff up or we don't know what we are talking about. And it's just, for us, it feels like more of a gradual progression. And I think a lot of what the guest would talk about predates AI.

Oh, yeah.

By many, many years.

By a lot.

And so, it's sort of like, for us, the fact that we can get together and eliminate certain bounds from code forever feels normal. But for the outside world, it feels like those guys are crazy. And that, to me, is the biggest find discovery from today. And all of this predates AI, by the way.

Oh, way predates AI. Listeners, to give you a sense of context, I have a 6-digit-long employee ID. Our guest who's about to come on the show has a 4-digit employee ID. And so maybe without any further ado, let's turn things over to one of our, not quite our most, but one of our most tenured guests of all time. Today we are joined by Christoph Kern, a principal security engineer here at Google, who says he's been here a very long time. Christoph, thank you for joining us today. I think a lot of people don't understand the extent to which Google and hopefully through the show they started to. We are a security company, and one of the ways in which we're a security company is the philosophy of secure by design and how that's permeated so much of our culture. And one of the things that I think has been talked about a little bit, but I want to talk about today, is how with the growth of AI-powered coding, how we're shifting from detecting vulnerabilities to eliminating classes of them. How do we do that? What does that mean? That sounds very I know it's a pie in the sky, but make that real for our listeners, please.

Yeah. So, it's interesting. I mean, this is something that we've actually been doing for, like, much longer than AI appearing on the horizon, right? This goes back probably 15 years, at least, this work.

Oh, that's a long time.

But it is now, I think, paying off more than ever due to the arrival of AI and AI being part of, like, the development lifecycle and ending up, you know, writing more and more code that Humans have less and less bandwidth to really review and understand in depth, right? And so being able to have high confidence that this code actually adheres to critical security invariants without having to read it is very, very important.

That's kind of amazing we can do that.

It always was important because, you know, humans make mistakes when they code and AI makes mistakes where they code. And so the situation hasn't actually changed that much.

It has just sort of scaled up in the urgency of, of So then pre-AI, what were we doing 15 years ago to eliminate classes of vuln? Make that real.

I mean, essentially we've realized that there is common classes of problems. You can see this, like the CWE, they have like these top 25. You've probably come across these, right? And if you look at them over the years, it's always the same usual suspects, right? There is classes of security defects that continue to appear over and over and over again. And we don't seem to be able to get rid of them using the approaches that we've taken before, which is testing and code review and static analysis and sort of basically all approaches that are centered around finding defects after they've been introduced, right?

Right.

The code gets written, and then afterwards, we try to find all the problems in it before we ship it. And that turns out to be a losing proposition for many of these classes of defects, which is really, you know, just there in the data. We can see that these problems have continued to appear and have been with us for, like, literally decades, right? So, what we've kind of, over the years, realized is that we kind of need to really understand why these problems occur and why these defects are getting introduced and what we can actually do about it. And the intuition is essentially that the root cause of these defects is in sort of risky programming constructs that we ask developers to use over and over and over again, very commonly, when they're writing application code.

Hmm.

And they're risky in the sense that if you're not very careful to ensure that this particular primitive's security precondition is, or correctness precondition is satisfied, you have some kind of undesirable or undefined behavior that often results in a security defect, right? So just to make it concrete, like every time in C or C++, you're dereferencing a pointer, as a programmer, it's your responsibility to ensure that when execution gets to that point, That pointer is valid, which means it is actually, you know, pointing to valid allocated memory of the right type, or that contains data of the right type. And you're in the range of the allocated region, and that memory hasn't been freed by some other part of the program. This gives you very many, you know, opportunities to—

There were a lot of conditions in there that you rattled off.

Yeah.

There's a lot of conditions. And the use of these risky statements is very common, right? Like, if you look at your C++. Look, there's pointers everywhere. There's array dereferences. In C, they've even made it potentially undefined behavior to add 2 integers, which is sort of like—

Sorry.

You're like, why? But I mean, there's good reasons, performance primarily, but in hindsight, this was probably not a good choice. Right. And so anytime you write code that uses one of these operations, it's your responsibility to make sure that its preconditions are satisfied and they're very complicated. And moreover, whether or not those preconditions are true, very often it's not something you can decide by just looking at the surrounding code. In some cases you can, because there's like an if statement right before that checks the precondition. But in many cases, that precondition gets established sort of by the program as a whole, right? So in particular, when you're looking at pointer lifetimes, whether or not that pointer is valid at this point in time kind of depends on not there having been some other part of the program that has deallocated since, right? And the owner, the sort of the module that might own that pointer that is responsible for the object's lifetime is not the code that you're usually working with, right? It's some other part of the program that you have to trust to not have to sort of pull out the rug underneath of you, right? And that is just extremely difficult, right? This is, like, an almost impossible task, which is why—

The way you're describing this makes me think, let's just not write C code anymore.

Exactly, right? And so, basically, what we sort of learned over the years is that, We're, in many cases, asking developers to write code in sort of an almost— it's a game you can't win, right?

Yeah.

It's an impossible ask, essentially. And so, the fundamental shift that we need to take, or have been taking, is to change that model into one where these risky operations are hidden away in safe abstractions, such that the developer when they're writing most of their application code, very rarely, if ever, encounters the need to use these kinds of risky operations. Rather, they get safe operations that are essentially safe in any valid use. So anytime you use the safe abstraction around the risky primitive, you're guaranteed by the type system and by the implementation of that primitive that the behavior is well-defined and nothing bad will happen. For reasonable definitions of bad, right?

So, while I want to ask a question about how this affects real developers, I wanna ask a follow-up to what Tim asked, actually. Here at Google, and again, my tenure is much shorter than yours, and I still have heard the line, eliminate the whole classes of vulnerabilities. And when I use the same line with people outside, they automatically assume that marketing invented it. But marketing did not invent it. Engineers did.

Yes.

And it gives me, like, a really odd vibe because when we say, hey, these are vulnerabilities, they have classes, people's code have many of those classes, our code does not have some of the classes. It's a very factual statement. And here at Google, and again, I'm not an AppSec person, I'm not tired of repeating this to people, but even I understand that to us, it's very factual. Here's the 10 classes. Here's Google code. Whatever number of classes are there, whatever number of classes is not there, they're eliminated. It's a very black and white, high contrast factual statement. Yet people on the outside kind of say, oh, yeah, yeah, yeah, yeah, this is marketing. You can't possibly mean that. But we absolutely do mean that, right?

Yes.

Yeah. And I think this is sort of a mindset shift that really actually took us a long time to get to, right? I forget exactly when it was. At one point in my career, it was like in the mid-2000s, I contributed to a book about software security. And I wrote the chapters on how to deal with some of these classes of vulnerabilities, like cross-site scripting and SQL injection. And I basically wrote the chapter exactly in the way that I just said we shouldn't do, which is to tell developers, here's like 16 different things you have to be careful about, and you just have to be careful about them always and not make a mistake. Earlier in my career, I had that same kind of thinking. And I think it does take a...

Yeah.

sort of a mindset shift to get to the point where you kind of realize that it's actually possible and very cost-effective and still ergonomic for developers to build developer platforms that really, like, actually just don't let you make that mistake without getting into your way, right? And that's sort of like navigating that trade-off here between imposing friction on developers that is irksome and something you really don't want and ensuring these correctness invariants with high degree of confidence is not that straightforward. Right. And it took, I think, the industry quite a while to get to. Right.

And why did it take us a while to get there? Isn't it kind of obvious that we should do this?

Yes.

No. Yeah. But it's obvious we should do it, but nobody, nobody is doing it.

How?

Yeah, it's about how.

Exactly. So the challenge is in doing it in a way that is both ergonomic for developers and performant. Right. So for instance, for memory safety, we've had memory-safe languages for a long time, but they all relied on garbage collection, which is a pretty significant overhead, right?

Yes.

And only in the last, like, 10 years or so, we have a production language that allows you to obtain the performance of an unsafe language like C and C++ with the safety of a memory-safe language, right? Of, like, similar safety to a garbage-collected language without the overhead. And that's something that really came out of research that was, like, in the 2000s, right? Like, there was a research language called Cyclone that I think was still being worked on in the mid-2000s that sort of laid some of the foundations of what made it into the Rust type system that actually lets you have these properties through static type checking and through properties of the type system itself. Nobody just knew how to do this in a, in a reasonable and ergonomic way in the, like, mid-'90s, right? It just didn't exist.

So the challenge was, it's actually just genuinely hard to do this technically. Yeah. Yeah.

And then making it work, even though conceptually— Yeah.

So it's conceptually hard and then—

Conceptually hard? Conceptually hard or operationally hard?

It's both, right?

Because I really don't like the problem that's conceptually hard, operationally hard. I prefer conceptually easy, operationally hard. But this is conceptually hard. Can I please go home?

Anton, you are home.

Well, I mean, you know, now we know, we've developed the patterns, right? But before these patterns existed, there was really no— nobody knew how to build a developer platform, a language that gave you the safety properties without being extremely cumbersome to use, right?

Mm-hmm.

And so, this basically is just something that took research in programming languages and experience in applying the learnings from research, from greenfield research into industrial languages. And then, you know, making it ergonomic, making it easy to use, making it friendly to use, having compilers with good error messages, and then, you know, like deploying this in a large organization with tens of thousands of developers without causing a lot of friction and overhead and cost. There's a lot of things we just had to learn over the years, right? And it took really, it was probably, like, at least a 10-year, 15-year journey to get to the point where we kind of feel like, yeah, now we have, we know how to do this for most of these common types of problems that we encounter in software development. And by the way, they're not all security problems, right?

Yeah.

There are other kinds of hard-to... hard to deal with defects, such as, you know, things that you sometimes would call like a Heisenbug, right? Like you have a concurrency defect that you just cannot reproduce in your developer environment, and it just happens every now and then in your production environment. Those are really painful to deal with. And they're a similar kind of problem, right? That is something that can just very occasionally occur in certain states of your software. And in this case, it maybe doesn't have security implications. It just has availability or reliability implications.

Mm-hmm.

But it's a similar kind of thing. It's a defect that is very, very costly to deal with once you ship it. And it is very desirable to design your software and build your software in ways that you can be quite confident are free of these defects. Right. And so I think that sort of approach to building software, there's just much more methodical and rigorous. It is actually something that you can genuinely call software engineering versus software craft.

Right, right, right, right.

And I think we're kind of basically learning how to do this in in cost-effective ways in sort of large-scale industrial-size development, right? Like, for things like avionics, this is probably something that's been used much more commonly. But that's a very, very different type of software. It's very expensive for the size and complexity of what it does, right? But it's also, you know, obviously important. And so people do it. But for something that is, let's say, consumer-facing software that's developed at a much higher velocity, at much larger scale, you just can't use these SDLCs that you would use for, you know, safety-critical software. It's just too— Too slow and cumbersome.

It's too expensive to program the space shuttle with 8 lines of code per developer per day.

Yeah. Yeah. There was this late— that came kind of from Heather, but it is stuck in my— it's burned in my brain, not just stuck in my brain. This whole 8 lines of code per developer per day thing is just like, I can't get out of it. I can't get rid of it.

Yeah. So, speaking of things that we can't get rid of, you talked about in this and elsewhere, security invariants, like all network traffic must be encrypted. How does Google, at industrial scale, decide which invariants are worth enforcing and which should be framework versus application? How do we do that at industrial scale?

Yeah. So, that's a very good question, right? So, one of the benefits that a large organization like Google has is that many of the things we do in the developer platform to improve the outcomes, the quality attributes of the software that comes out of it, get amortized over a large number of applications, right? So we have thousands of web front ends. And so if we build safety and correctness, security, privacy, reliability, and internationalization primitives into the developer platform for web applications, the cost of that gets distributed over thousands of instances of those applications. So that's one way of making it very cost effective. And then that leads into the decision of what you actually lower into the platform. There's sort of this concept of shift down, right? Like what is most profitable? And there's sort of 2 aspects of it, right? One is it's a property that, that is relevant to the class of applications, right? So a property that is relevant to all web frontends, irrespective of exactly what they do. And it is something that is actually implementable in the platform level that can be sort of, can actually be shifted down into the platform. Some properties are much harder.

So something that's both worthy and possible.

Exactly, right? And so, then when you have both, then it's a— and then also, you know, it matters whether or not it's a very common problem, right? Or a very important problem. It could be uncommon, but very bad when it happens, which makes it important. Or it could be a thing that you just deal with very frequently, and therefore, and not that bad, but it still makes it worthwhile to address and get rid of, right? As a— or, well, I mean, it's the getting rid of the problem is sort of the ensuring of the invariant that implies that the problem is not there, right? There's sort of two sides of it. Right.

One of my teams early in Google was an early adopter of the BOK framework. And at first, we were totally baffled by it and found it to not be especially ergonomic. And then after about 2 weeks, we were all like, whoa, this solves so many things we no longer have to even keep in our mental swap space. This is so worthwhile. And so, it must be really interesting to work with teams early in that journey.

Exactly. Yeah. And so I think what matters there is whether or not the developer platform, and I like to kind of use an expansive view of that, right? It's not just the production platform, it's the tech stack, like application frameworks, like what you mentioned, Bok, and the toolchains, the code repository, your CI/CD pipeline, all those things, they all work together to inform the outcomes in terms of quality and correctness and scalability of the software that you're building. Right. And so what is—

Yeah.

I think very important is whether or not the platform that you're using is actually tuned to the type of application you're trying to build and you're not chafing at it. Right. And so the trick is basically to find ways to build platforms that actually match the requirements of the types of applications that people want to build. And, you know, hopefully you don't have too many variations of this, right?

So Google can amortize the cost of developing this stuff over thousands of teams building software. How can not Google, who are most of our listeners, think about adopting some of this? How should a random company that doesn't have its whole business building software but actually builds pharmaceuticals or tractors or financial products think about something like this?

Yeah. So this is interesting. I think there is some cause for optimism, I think, because—

Does it involve 2 letters, AI, or not?

Well, it does and it doesn't, right? So the interesting thing is, Okay. I'll get to the AI, right? I think what has happened is that this way of thinking about how you build software in an industrial setting at scale and reasonable cost-effectiveness has really evolved. And we now see large adoption, for instance, of Rust in the— it has a very vibrant community that really cares about these concepts that I talked about. They're like, to me, they seem like they're very well understood in the Rust community. They're basically like, yeah, this is obvious, right? You don't need to explain this to me. And in fact, I'm very happy to see that the things we came up in terms of how we, for instance, address vulnerabilities and classes of problems for web applications, they really have a really nice conceptual analogy to how the Rust community thinks about memory safety. And they're really, if you look at it from a certain perspective, they're very, very relatable. And so these have evolved independently. So it's a kind of a nice support for the idea that we're sort of on a good track here. And then, you know, for instance, for web security, we've actually been able to externalize some of the work we've done originally internally in our monorepo, these concepts of using types to ensure that injection sinks can only be reached by values that are actually safe to send there is now built into the web platform. So we were able to work with browser vendors. We started with Chrome, obviously, because it's part of Google and we have good relationships there, but this has been now adopted by all the major browsers. And you can use this as a developer to effectively harden your application and make sure— basically gain very high confidence that your web application is free of cross-site scripting. And it embodies these ideas. So this is now part of the web platform. You can use it. And many of the popular web application frameworks like Angular and React are compatible with it or interoperable with it. You can use it with those. So that's an example. So I think you can go a fairly long way by using one of those stacks for your web front ends and using a memory-safe language for your server-side infrastructure. And there are, I think, more and more elements of this that are becoming readily available. One interesting thing is, you know, we've thought about like, why don't we actually publish an open source version of something that we've built here? And I think it has not happened and may not work because of what I said earlier.

Too hard.

It's that these platforms need to be tuned to exactly the type of your style of application development in your organization. Otherwise you'll chafe at it, right? So they basically, almost by necessity, they achieve their quality outcomes by being opinionated. And the opinions of the platform need to align with the needs and the opinions of the developer community. And so you probably actually would find it very difficult to build an opinionated, secure developer platform that works for everybody. What is, however, I think feasible is to have platform components like a web frontend framework, for instance, that are combinable in opinionated ways. and achieve these outcomes, but it can be done. You basically kind of need a small platform team that does this for you, right? Like, and this could just be like a slice of one person if you're a small shop, but you need somebody to think about how do I assemble a developer platform for my needs, for my organization's needs that has, you know, the opinions that, that imposes the opinions that achieve these outcomes.

On that opinionated platform front, it's very interesting to compare the trajectory of something like App Engine, which is highly opinionated with, say, Kubernetes, which is also highly opinionated, but looser in how you can interact with it, and how those two have, for external adoption, had very different trajectories.

I think opinionated, though, triggers some people, right? Because both positively and negatively, right? I mean, people show up and say, if this code has a vuln, my compiler refuses to work for you, kind of stuff. Like, referring to your point about security should be, like, defects and code shouldn't compile if it's not secure. I mean, it's, like, strongly opinionated down to basically saying, I'm a compiler, I decide what you do. But at the same time, people say, I want freedom. I want to be a, what is it, software artist. Not the software engineer, but not the software craftsman. I want to be a software artist.

Yeah.

How does it work? It sounds kind of a bit of an empathy, like, it's a risk. The high opinionated stuff occasionally looks like it's like hubris because we know better. And I am torn about this in many other areas like SecOps or vulnerability management. Maybe you can save us by pointing out how do you balance opinionated with empathy?

Yeah, exactly. I think that's really important, right? And this again comes down to the platform has to have opinions, but they need to be opinions that don't conflict with what developers are actually trying to do so that Most of the time, the thing that comes naturally actually agrees with those opinions and does not, and then it just works. And it also is secure and safe and scalable and everything, right? And so I think a lot of that is around the feedback loops between those platforms and the developers and the problems they actually run into. And it sort of requires to have a friendly, open community if it's a project that is, you know, out there in the community or internally.

Yeah.

One thing we've done, I think one pattern that I really like that we've adopted in our team is that the engineers that build these opinions into frameworks and build these constraints into frameworks actually serve as the first-line customer service to developers.

Mm-hmm.

And so whenever there is friction points, they will know immediately because all of a sudden they get like an uptick in the number of questions about a thing. And then that sort of forms a feedback loop where we see, okay, this is kind of a common friction point. So maybe we should make a nicer API for that, right? Or a special API, or we should maybe, or maybe this constraint is actually not that valuable because the issue that it prevents from happening is not that severe and it happens very rarely. So maybe we just let it go, right? Because it's not worth it. And so you kind of like iterate over time on sort of what the platform's opinions should be and how strictly they should be enforced and exactly how they're phrased, so to speak, right?

I like most of it. I like 95% of that. But I'm a little— I want to ask about one point you made in the beginning about how the opinionated should align with developer. What term did you use? Goals or choices?

Yeah, or their practices, what comes natural to them in some sense.

No, no, no, no. But okay, fine. Then I'll argue. Because, okay, I'm a really bad developer and my natural inclination and the past experience is that I should put passwords in PHP code. So, I think that if you go with my natural inclinations, the results would be quite bad. So, I want opinionated— I don't want opinionated, but I probably should be governed by opinionated tool that says, dude, don't put passwords in the PHP code. But that's— how do you deal with that?

Yeah. So, I think at some point, certain choices that a developer might make essentially have externality costs, right? Like, if you do that, it might work for you because it's easier and you might ship your task, like, a day earlier perhaps, but you're essentially essentially taking on a significant cost of externalities for your org. And so at some point, it's not your decision, it's your org's decision. And so we've actually kind of seen this, that at some point, the leaders of organizations see that a particular platform in the aggregate provides good trade-off between development velocity and quality outcomes. And then they essentially just mandate the platform's opinions for their org. And at some point, this is sort of like, It's perhaps not the individual developer's decision because they're building enterprise software that has to satisfy certain quality attributes. It has to be secure, it has to be scalable, it has to be reliable. And how you achieve that may kind of conflict with your natural instincts. But at that point, perhaps, I mean, you're doing this not for fun, you're doing this for work, right? So, you just have to get with the program at some point, right?

I think a lot of people over-rotate on what's natural. I like to joke that I'm gonna put a piece of tree bark into the floppy disk Slotify is being really natural about things. So, unfortunately, we're coming up on time, and I'm gonna do a sneaky thing. I'm gonna ask a question as our traditional closing question, which is, if I'm a Fortune 500 CISO or head of development listening to this, my core competency is not building software, but we're enabling the business by doing it, how do I prove to my leadership that secure by design is not a cost, but actually, in the long run, is going to be an upside for us? Like, what are the metrics I want to track for that? How do I tell that story? And then 2, second part of our traditional closing question, do you have recommended reading for our audience?

Yeah. So I think the cost equation is interesting. I think there is data starting to come out that corroborates this. So for instance, the Android team, they posted some blogs and I think articles about their experience in shifting from memory-unsafe languages to memory-safe languages, from C and C++ to Rust and Java, primarily. And so, they've seen, obviously, a very— I mean, not obviously, they have shown the evidence that this has resulted in a very significant drop in the reported vulnerabilities that they're getting. But it also has shown, they have experienced significant gains in productivity that are measured in, for instance, the rate at which changes, commits, get reverted and the time it takes to code review and so on. Right. So, and also self-reported, you know, just developer productivity metrics.

Huh.

So there are certain, some productivity metrics that actually go up when you're switching to a language that is at first sight, perhaps harder to use. Right. Because it, I mean, Rust imposes more constraints than C++. It lets you not make all these mistakes and you have to kind of demonstrate to the compiler that it is correct, that your code is correct, which takes some getting used to. But once you get the hang of it, there's this whole bunch of problems you don't have to deal with anymore because they won't happen. They just don't even compile, right? And so you're, it might feel like you're making more progress because you're not dealing with the compiler all that much, right? But then you actually end up in practice shipping more bugs that you have to deal with later and rollback CLs and so on, right? And so some of that data is, I think, starting to appear. But again, it is kind of, quite, I think, specific to organizations and how organizations work. And so at some point you probably have to start making experiments and just demonstrate the data in your context, right? And demonstrate the experience in your context. And reading, um—

Well, you got one in there, which is the Android blogs about this stuff. So you were yourself sneaky and smuggled the reading into the tip. I love that.

That's very good.

There is a, uh, so a couple of years ago we've written a book called Secure and Reliable Systems, which has some chapters that touch on this. And then I guess shameless plug, I recently, a couple of months ago, I had an article in Communications of the ACM that talks about safe coding, which is basically this. So the actual mechanics, like how does this work? How does this software design practice actually look like? And also that touches a little bit on— so when we're talking about invariants, right? The thing about invariants is it basically, it has like an implicit universal quantifier in there. So you kind of like hint you need to use logic to actually demonstrate this property always holds. You can't do it by testing, right? Because testing just gives you a few cases out of the astronomically large number of possible behaviors of your program. So in that article, I try to make a connection to formal reasoning about programs, to rigorous logic-based reasoning about programs. And there is actually, I think, some good parallels to the practice that is used in in Rust as well in our sort of type-based approach to preventing web vulnerabilities or injection vulnerabilities in general.

So, wow.

So that's kind of a bit more technical, but this is sort of like the nuts and bolts of how this actually works.

Oh, that's very cool. Uh, Christoph, thank you so much for joining us today. This was an incredibly deep look at how Google got to be more secure and what went into that. This is phenomenal.

This is just epic. Sorry. This is like, we had high expectations, but they were exceeded.

This is super rare.

It was great talking to you.

Yeah, yeah.

I mean, Gemini called you a titan, so it must be— Yeah, I don't know about that.

And now we are at time. Thank you very much for listening, and of course, for subscribing. Please subscribe so you can get new episodes piping hot. And if you love our content, please drop us a review on your platform of choice.

You can find this podcast on YouTube, Apple Podcasts, Spotify, or wherever you get your podcasts. Also, you can find us on our website, cloud.withgoogle.com/cloudsecurity/podcasts.

You can argue with us on the Google Cloud Security Community site, googlecloudcommunity.com.

You can follow us on X, x.com/cloudsecpodcast. Tweet at us, email us, argue with us. And if we like or hate what we hear, we can invite you to the next episode.

See you on the next episode of the Cloud Security Podcast by Google.

Are you ready?

I'm ready.