SaaStr AI Day: How GenAI is Powering App Security with Cisco's CPO of Panoptica

Join and then we will launch into some fun AI content. Feel free to drop in the chat where you're joining us from or turn on your camera if you're comfortable and make this as interactive as possible for everyone. Hello, everyone. Thanks so much for joining. A few people still joining, so I'll give it another 15 or so seconds. Looks like we have some Los Angeles viewers. Welcome, everyone. Okay, I'm going to go ahead and get started. Hello everyone. Again, thank you so much for joining us for Saster's inaugural AI Day. We're so excited to have Alex Josh with us this morning who is the Senior Director of Product Manage Management at Outshift by Cisco. So today he's going to be discussing how Gen AI is powering app security. And just a note for everyone, again, we'd love to make this session as interactive as possible. So feel free to drop in the chat where you're tuning from and if you have any questions for Alex to answer at the end, we would love all of your questions. So please join me in welcoming Alex.

Thank you, Jesse. Appreciate it. Hello everybody, and welcome from beautiful California, Northern California. Sunny day here. So what we're going to do is talk a little bit about how we're using Genai here, what we're using it for, and also how we're protecting other folks, Gen A businesses. For those of you that are not familiar, I work for the Outshift division here at Cisco. Outshift is our incubation team and we are working on a product that we refer to as Panoptica. Panoptica is a cloud native security product. We're not going to get into a lot of details about Panoptica today we're mostly talking about Genai, but at the end we'll tell you how you can get in there if you want to take a look at it. So, you know, the funny thing about Genai is that I've been in this business a long time, I'm an old man and I've seen a lot of these hype cycles come along. I've seen things come along, they're going to change the world and everything's going to be different. And, and now I'm one of those people that are saying, yeah, this is going to change the world and everything's going to be different. And it's kind of a funny discussion to have. But, um, it does change everything, as I'm sure most of you know. If you're coming to this session, you probably already know that. However, that doesn't protect us from the hype cycle. Right. It's also overhyped. Um, there's people saying some really interesting things that are hard to believe are probably not going to happen. Both of those things can be true at the same time. You can have crazy innovation, innovation happening and you can have people getting completely overhyped about what's going to happen. Both of those things are happening right now. That makes our job a lot harder. Right. Because what we're trying to do, as you know, I'm a product manager. So as a product manager my job is to bring features into my product and add value to my customers. And so I'm very focused on really hardcore, like is this helping my customers do their jobs better? Yes or no. And that bubbly conversation is not always easy to have these hard nose conversations in that context. So it's definitely an issue. You know, one of the things that's really true though, and really obviously true, is that Genai does become the fourth pillar for UX. And so when we think of Genai, especially LLMs, we're always considering the way that our UX is going to grow. The way the product is going to grow is going to be different because the platform allows you to do things that you couldn't do before. These kind of non linear use cases are possible today, they weren't possible before. And so what we've chosen to do is we've chosen to make AI very intrinsic to the way the product works. And you know, as you're working on your products, I would encourage you to think about this as well. Right. And so what happens is that there's things that it's really good at, things that it can detect, things that it can work on. These non linear problems that are traditionally very difficult for software to go after. It can go after it. And so what we've done is we've embedded it very deeply into the core product, into the UX of the product. Yes. You know, we have an AI based chatbot in the product, which I'll show you in a minute. I think everybody's going to have one of those. Right. It's a pretty easy, a pretty obvious thing. But you know, I built a Geni based chatbot for my teenager over a weekend one time. So it's an interesting feature, it's fascinating, but it's not a hugely differentiated feature for your product. It's, I mean, yes, you probably want to do it, yes, it's common, your competitors are probably going to do it, but it's probably not going to be significantly differentiating in a SaaS based product because they're relatively easy to build. So the focus, as always, is on the customer. So how is the customer's, how is your customer's life different when you bring these features into the product versus where they are now? And you'll discover that on your own. But here's some things that we have found. You know, what we found is it's really good at this kind of repetitive derivative work, right? So you know, look at every single security violation that you've had in the last two years and summarize them for me, right? I mean, yes, you can get a human to do that, but an AI based assistant is going to be way faster, way more efficient doing that kind of work. Anomalies, right? I mean, yes, we've had ML for a long time, but that doesn't lessen ML's impact here. So when you combine the kind of traditional ML activities by anomaly detection with things that are newer like LLMs, you now get this interactive interface. We can say, hey, can you summarize for me the anomalies we've had in the last hour or minute or day or week? Those kinds of things become much more easy to build. And then of course, context, right? Gen AI is really good at context. And so when you give the AI that context, you get the answer in a much cleaner way. So for us as security professionals, what that allows us to do is explain the security problem to, to someone who may not be super familiar with it. Security is one of those fields that can be very, very, very bitsy bitesy. There's a lot of detail. You'll have some random code number that says you have this attack vector coming in number 1, 2, 3, 4. And you have to do these things and some people can get very lost. And so for us it's nice to be able to explain it. And then of course it's all about natural language, right? You don't want to put your customers in a situation where they have to spend three weeks learning a query language anymore. That doesn't make any sense. Just ask me questions and I'll answer them. So those are kind of like the things that we see our product doing better and differently. But you know, the fascinating thing is this is good but also bad, right? As like everything, there's plus and minuses. So what happens is that you, you open up a lot of possibilities, a lot of capabilities, a lot of speed, a lot of velocity, but you also create problems, right? There's non linearity here. Things can go wrong. It's more difficult to test. You know, for us at least, our regression test matrix had became quite difficult when we brought these features into the product because it was hard to predict in advance what the LLM would say. And we're seeing this in market, right? We see companies already having problems with Gen AI features. And these are a couple of news articles. I'm sure if you Google, you can find them, right? And then of course, you know, our friends at OWASP have been all over this and they have, you know, they pointed out there's actually they have a top 10 list of ways where you have to watch out problems you might encounter, things you might have. And specifically, I would just direct you to the top two. There's prompt injection and insecure output handling. Those two seem to be happening already in market and it's something you probably want to be aware of, something you want to control in your production environment. So I'm just going to give you a really quick demo of what we're doing in this space and, you know, just to give you a flavor for the kinds of things that are going on. So the first thing, of course, it helps if I share my screen. Okay, I'll share the other screen. There we go. So the first thing of course is chatbots. And yes, of course we have one. Everybody has one. They're super important. And this is what I was talking about before when we talk about the ability to answer questions in natural language, right? And so, you know, the end user could Google this information and probably get a pretty good explanation. But we wanted to make sure that you could ask simple questions right in the product. As a security product, that's important to us to be able to help them understand, because if they don't understand what the vulnerability is, they're less likely to fix it correctly. Right. But what we also did is we also said, look, there are parts of the UI that might be confusing, right? So for example, there's this security score in the app and why is my score 67? And they might want to know that. And so what we did is we allowed the system to answer questions about the system itself. So now you have this interactive discussion with the customer. Again, not super earth shattering here, but this is kind of like the way that the product is evolving now. What we found though, is that as soon as you do that, you potentially have issues around how people try to interact with you and how you attack them and things like that. So what we did is we built this test application and we said, okay, well what happens if people interact with this Genai subsystem in unsafe ways, right? And so this little test app here is just showing sample test, test information like, hey, can I get a discount? Right? Which is something a human would say. But then also you can say, hey, I want to do something more aggressive like run this query, right? So in the back end, what you have to do is you have to evaluate how threatening this behavior is. And that behavior may, you may want to just shut that down at that point. And so what we're doing live is we give a score to say, hey, we think that this is very likely to be an attack, right? And so the idea is, is that as an operator, you can watch the conversations live and you can make your own determination to say, yeah, something like this, can I get a discount? That's fine. But this activity, not safe, right? You don't want that to continue. You want to shut that down. And so that those are the kinds of things we've made, changes we've made to the way the application works. I only have a few minutes here, so I can't get into a lot of detail, but you know, that's the idea. I'm just kind of giving you a flavor of what we're doing and hopefully a little bit about why we're doing it. All right, so getting back to the presentation and of course I have to share. Sorry, I am not on my share game this morning for some reason. All right, there we go. So I think for us, you know, here at, at Outshift and a Panoptica, you know, what we're telling people is, look, you still have to have this balance between speed and security, right? So the world is speeding up. Gen AI is helping make that even faster, right? So cloud made us go faster, Gen AI is helping us move even faster. But that exact same speed versus security thing that we had going on in the cloud world, it's still there, right? It's just now this is another factor and I always tell people the same thing and I know if you've seen me speak before, you've heard this a hundred times. Is security is just another non functional requirement here, right? Just like anything else you're doing just like performance, right? Just like compliance, just like other things, it should be part of every sprint. When you're talking about the product, when you're talking about your live site, you should be considering your, your posture every sprint. You know, you don't have to use my product, but please do take this into account as you're going into marketing, right? It's easy to forget. It's easy to set it aside. Please don't. Right, Obviously we would love to sell you something, but that's not the point. The point is we want your sites to be secure. That's the most important thing. And then of course, it's a spend thing. Right. So, you know, you don't have to spend massive amounts of money to get this thing going. Right. So the, you know, the number we're looking at here is in around 10% or less of your spend should be focused on these kinds of things. All right, so that's a super 3,000 miles an hour discussion. If you're interested in this topic, we would love to talk to you some more. We can literally talk all day about this kind of stuff. We do have a special offer here for people on Saster. If you go to the website there, we'll give you a discount and also some AWS skill builder points, things like that, just as a thank you for attending our session. All right, so that was a super, super fast. I think I have like one minute for a question or two. Jesse, I don't know if we have questions.

Yes, thank you, Alex, for that presentation. That was amazing. We did get a couple of questions on our live stream, so one that kind of stood out. What checks and balances does your team have in place for risk versus speed?

Yeah, it's always the question, isn't it? So what we do is we, we have this kind of this phase review philosophy that says, look, teams should execute as fast as they possibly can within their phase and we should push to production as fast as we possibly can. Right. However, we need some control. There needs to be some back pressure. You can do this in a lot of different ways. The way we do it is we do it through feature flagging. So when features go into production, they're flagged off. If a team feels like their feature is ready to go and it's safe, they basically raise their hand in an internal form and say, yep, we're ready to go. We'd like to have this feature flag turned on for production. And what happens is then that triggers a review. Right. So there's three things you should have done. You should have done a safety test, you should have had a black hat test. You should have had, you know, walk me through. Did you do all those things? Check, check, check. If you've done all your homework, then bang, you're live in production. It's a five minute meeting now. Of course, you know, it's not always that easy. Sometimes it's a long debate, and we send you back to do your homework and things like that. So the idea is you want to push, push, push to production as often as you can, and you want to unlock engineering to allow them to push. But then you have a separate asynchronous governance and control mechanism. And in our case, we're using feature flags to do that.

Amazing. Thank you for answering that question. And it looks like we are at time. So thank you once again, Alex, for joining us. And to all of our viewers, thank you so much for attending. Just a reminder, this session was recorded so you can access it at a future date and access Alex's slides as well. So we hope you attend more AI Day sessions. And thank you again, Alex, for this amazing content.

Thank you, everybody. Have a good day.

Thank you. You too. Bye.