SaaStr AI Day: How GenAI is Powering App Security with Cisco's CPO of Panoptica

join, and then we will launch into some fun AI content. Feel free to drop in the chat where you're joining us from, or turn on your camera if you're comfortable, and make this as interactive as possible for everyone. Hello everyone, thanks so much for joining. A few people still joining, so I'll give it another 15 or so seconds. Looks like we have some Los Angeles viewers. Welcome everyone. Okay, I'm gonna go ahead and get started. Hello everyone again. Thank you so much for joining us for SaaStr's inaugural AI Day. We're so excited to have Alex Josh with us this morning, who is the Senior Director of Product Management at Outshift by Cisco. So today he's going to be discussing how GenAI is powering app security. And just a note for everyone, again, we'd love to make this session as interactive as possible. So feel free to drop in the chat where you're tuning from. And if you have any questions for Alex to answer at the end, we would love all of your questions. So please join me in welcoming Alex.

Thank you, Jesse. Appreciate it. Hello, everybody, and welcome from beautiful California, Northern California. Sunny day here. So what we're gonna do is talk a little bit about how we're using GenAI here and what we're using it for, and also how we're protecting other folks' GenAI businesses. For those of you that are not familiar, I work for the Outshift division here at Cisco. Outshift is our incubation team, and we are working on a product that we refer to as Panoptica. Panoptica is a cloud-native security product. We're not going to get into a lot of details about Panoptica today. We're mostly going to be talking about GenAI. But at the end, we'll tell you how you can get in there if you want to take a look at it. So, you know, the funny thing about GenAI is that I've been in this business a long time. I'm an old man and I've seen a lot of these hype cycles come along. I've seen things come along, they're going to change the world and everything's going to be different. And now I'm one of those people that are saying, yeah, this is going to change the world and everything's going to be different. And it's kind of a funny discussion to have. It does change everything, as I'm sure most of you know, if you're coming to this session, you probably already know that. However, that doesn't protect us from the hype cycle, right? It's also overhyped. There's people saying some really interesting things that are hard to believe or probably not gonna happen. Both of those things can be true at the same time. You can have crazy innovation happening and you can have people getting completely overhyped about what's gonna happen. Both of those things are happening right now. That makes our job a lot harder, right? Because what we're trying to do as, you know, I'm a product manager. So as a product manager, my job is to bring features into my product and add value to my customers. And so I'm very focused on, you know, really hardcore, like, is this helping my customers do their jobs better? Yes or no? And that, you know, bubbly conversation is not always easy to have these hard-nosed conversations in that context. So it's definitely an issue. You know, one of the things that's really true though, and really obviously true, is that GenAI does become the fourth pillar for UX. And so when we think of GenAI, especially LLMs, we're always considering the way that our UX is going to grow. The way the product is going to grow is going to be different because the platform allows you to do things that you couldn't do before. These kind of non-linear use cases are possible today. They weren't possible before. And so what we've chosen to do is we've chosen to make AI very intrinsic to the way the product works. And, you know, as you're working on your products, I would encourage you to think about this as well, right? And so what happens is that there's things that it's really good at, things that it can detect, things that it can work on, these nonlinear problems that are traditionally very difficult for software to go after. It can go after it. And so what we've done is we've embedded it very deeply into the core product, into the UX of the product. Yes, you know, we have an AI-based chatbot in the product, which I'll show you in a minute. I think everybody's gonna have one of those, right? It's a pretty easy, a pretty obvious thing. But, you know, I, I built a GenAI-based chatbot for my teenager over a weekend one time. So it's an interesting feature. It's fascinating, but it's not a hugely differentiated feature for your product. I mean, yes, you probably wanna do it. Yes, it's common. Your competitors are probably gonna do it, but it's probably not gonna be significantly differentiating in a SaaS-based product because they're relatively easy to build. So the focus as always is on the customer, right? So how is the customer's, how is your customer's life different when you bring these features into the product versus where they are now? And you'll discover that on your own. But you know, here's some things that we have found. You know, what we found is it's really good at this kind of repetitive derivative work, right? So, you know, look at every single security violation that you've had in the last 2 years and summarize them for me, right? I mean, yes, you can get a human to do that, but an, an AI-based assistant is gonna be way faster, way more efficient at doing that kind of work. Anomalies, right? I mean, yes, we've had ML for a long time, but that doesn't lessen ML's impact here. So we, when you combine the kind of traditional ML activities like anomaly detection with things that are newer like LLMs, you now get this interactive interface where you can say, hey, can you summarize for me the anomalies we've had in the last hour or a minute or day or week? Those kinds of things become much more easy to build. And then of course, context, right? GenAI is really good at context. And so when you give the AI that context, you get the answer in a, a much cleaner way. So for us as security professionals, what that allows us to do is explain the security problem to someone who may not be super familiar with it. Security is one of those fields that can be very, very, very bitsy-bitsy. There's a lot of detail, you know, you know, you'll have some random, you know, code number that says, you know, you have this, you know, this attack vector coming in, number 1, 2, 3, 4, and you have to do these things. And some people can get very lost. And so it's, for us, it's nice to be able to explain it. And then of course it's all about natural language. Right? You don't want to put your customer in a situation where they have to spend 3 weeks learning a query language anymore. That doesn't make any sense. Just ask me questions and I'll answer them. So those are kind of like the things that we see our product doing better and differently. But you know, the fascinating thing is this is good but also bad, right? As like everything, there's pluses and minuses. So what happens is that you you open up a lot of possibilities, a lot of capabilities, a lot of speed, a lot of velocity. But you also create problems, right? There's nonlinearity here. Things can go wrong. It's more difficult to test. You know, for us at least, our regression test matrix had became quite difficult when we brought these features into the product because it was hard to predict in advance what the LLM would say. And we're seeing this in market, right? We see, companies already having problems with GenAI features. And these are a couple of news articles. I'm sure if you Google, you can find them, right? And then of course, you know, our friends at OWASP have been all over this and they have the, you know, they pointed out there's actually, they have a top 10 list of ways where you have to watch out, problems you might encounter, things you might have. And specifically, I would just direct you to the top 2 there. Prompt injection and insecure output handling. Those two seem to be happening already in market and it's something you probably wanna be aware of, something you wanna control in your production environment. So I'm just gonna give you a really quick demo of what we're doing in this space and, you know, just to give you a flavor for the kinds of things that are going on. So the first thing, Of course it helps if I share my screen, eh? Okay, I'll share the other screen. There we go. So the first thing, of course, is chatbots. And yes, of course we have one, everybody has one, they're super important. And this is what I was talking about before when we talk about the ability to answer questions in natural language, right? And so, you know, the, the end user could Google this information and probably get a pretty good explanation, but we wanted to make sure that you could ask simple questions right in the product. As a security product, that's important to us to be able to help them understand, because if they don't understand what the vulnerability is, they're less likely to fix it correctly, right? But what we also did is we also said, look, there are parts of the UI that might be confusing. right? So for example, there's this security score in the app and why is my score 67? And they might want to know that. And so what we did is we allowed the system to answer questions about the system itself. So now you have this interactive discussion with the customer. Again, not super earth-shattering here, but this is kind of like the way that the, the product is evolving now. What we found though is, is that as soon as you do that, you potentially have issues around how you— people try to interact with you and how you attack them and things like that. So what we did is we built this test application and we said, okay, well, what happens if people interact with this GenAI subsystem in unsafe ways, right? And so this little test app here is just showing sample test information like, hey, can I get a discount, right? Which is something a human would say. but then also you can say, hey, I wanna do something more aggressive, like run this query, right? So in the backend, what you have to do is you have to evaluate how threatening this behavior is, and that behavior may, you may wanna just shut that down at that point. And so what we're doing live is we give a score to say, hey, we think that this is very likely to be an attack, right? And so the idea is, is that as an operator, you can watch the conversations live And you can make your own determination to say, yeah, something like this, I can, I get a discount, that's fine. But this activity, not safe, right? You don't want that to continue. You want to shut that down. And so that, those are the kinds of things we've made changes, we've made to the way the application works. I only have a few minutes here, so I can't get into a lot of detail, but you know, that's the idea. I'm just kind of giving you a flavor of, of what we're doing and hopefully a little bit about why we're doing it. All right, so getting back to the presentation. And of course I have to share. Sorry, I am not on my share game this morning for some reason. All right, there we go. So I think for us, you know, here at at Outshift and at Panoptica, you know, what we're telling people is, look, you still have to have this balance between speed and security, right? So the world is speeding up. Gen AI is helping make that even faster, right? So cloud made us go faster. Gen AI is helping us move even faster. But that exact same speed versus security thing that we had going on in the cloud world, it's still there, right? It's just now this is another factor. And I always tell people the same thing, and I know if you've seen me speak before, you've heard this 100 times, is security is just another non-functional requirement here, right? Just like anything else you're doing, just like performance, right? Just like compliance, just like other things. It should be part of every sprint. When you're talking about the product, when you're talking about your live site, you should be considering your posture every sprint. You know, you don't have to use my product, but please do take this into account as you're going into market, right? It's easy to forget, it's easy to set it aside, but please don't. Right? Obviously we would love to sell you something, but that's not the point. The point is we want your sites to be secure. That's the most important thing. And then of course it's a best spend thing, right? So, you know, you, you don't have to spend massive amounts of money to, to get this thing going, right? So the, the, you know, the number we're looking at here is, you know, around 10% or less of your spend should be focused on these kinds of things. All right, so that's a super 3,000 miles an hour discussion. If you're interested in this topic, we would love to talk to you some more. We can literally talk all day about this kind of stuff. We do have a special offer here for people on SaaStr. If you go to the website there, we'll give you a discount and also some AWS Skill Builder points, things like that, just as a thank you for attending our session. All right, so that was a super, super fast, I think I have like 1 minute for a question or 2. Jessie, I don't know if we have questions.

Yes, thank you, Alex, for that presentation. That was amazing. We did get a couple of questions on our livestream. So one that kind of stood out, what checks and balances does your team have in place for risk versus speed?

Yeah, it's always the question, isn't it? So what we do is we have this kind of this phase review philosophy that says, look, teams should execute as fast as they possibly can within their phase, and we should push to production as fast as we possibly can, right? However, we need some control. There needs to be some back pressure. You can do this in a lot of different ways. The way we do it is we do it through feature flagging. So when features go into production, they're flagged off. If a team feels like their feature is ready to go and it's safe, they basically raise their hand in an internal forum and say, yep, we're ready to go, we'd like to have this feature flag turned on for production. And what happens is then that triggers a review. Right? So there's 3 things you should have done. You should have done a safety test. You should have had a black hat test. You should have had, you know, blah, blah, blah. Walk me through, did you do all those things? Check, check, check. If you've done all your homework, then bang, you're live in production. It's a 5-minute meeting. Now, of course, you know, it's not always that easy. Sometimes it's a long debate and we send you back to do your homework and things like that. So the idea is you wanna push, push, push to production as often as you can, and you wanna unlock engineering to allow them to push. but then you have a separate asynchronous governance and control mechanism. And in our case, we're using feature flags to do that.

Amazing. Thank you for answering, uh, that question. And it looks like we are at time. So thank you once again, Alex, for joining us. And, uh, to all of our viewers, thank you so much for attending. Uh, just a reminder, this session was recorded, so you can access it at a future date and access, um, Alex's slides as well. So we hope you attend more AI Day sessions, and thank you again, Alex, for this amazing content.

Thank you, everybody. Have a good day.

Thank you. You too. Bye.