Testing AI Vibe Coding: Stop Vulnerabilities Early with Sarit Tager

In a land of testers far and widely journey seeking answers, seeking skills, seeking a better way O the Test Illumination Testing Podcast Guiding testers with our nation awesomeness from ancient realms to modern days, they lead the way. Oh, the Testicle Ovation Testing Podcast.

Hey. Today's episode goes deep into something I think is becoming unavoidable for automation engineers, and that is how AI generated code creates new risk and with that, new responsibilities. Whether using cursor, playwright, mcp copilot or internal AI agents, the way you build, test and produce code is changing fast. And that's why I'm really excited to have Saret Tiger joining us from Palo Alto Networks, helping break down what testers and automation engineers need to know. Things like how vulnerabilities get introduced and how you can use guardrails to protect you before the code ever hits your repo. If you're building automation frameworks, using AI to generate code, or working in pipelines that ship fast, this episode is packed with insights that directly affect your day to day work. You don't want to miss it. Listen up. Hey Sarah, welcome to the Guild.

Hey Joe, nice to be here.

Awesome to have you. Well, I'm really excited to have you on the show. You have a lot of experience and you work for a really cool company. So I think before we get into the topic, I think we're going to cover vibe, coding, security, all those things that are really happening right now. Maybe a little background. How did you become like the VP of Product at Palo Alto Networks?

So I will tell a bit about my product and a fun fact where I started from. So I'm the VP Product for Application Security within Palo Alto as part of the Cortex Cloud family and we kind of use everything from code scanning to pipelines to ASPM to application security, posture management. Super exciting subjects and very, you know, very evolving one. A lot of AI is in place. The fun fact is that I came to this by being a developer first and then VP engineering. And I always say that I came to the application security and the cloud security area in a mission to make it better for developers and for appsec practitioners and for security practitioners generally. I do remember a lot of nights sitting and trying to fix things security issues as a developer and I think this is part of what made me become a a product manager in this area. And yes, I've been working for like a variety of companies from network side of things to cybersecurity of images and even there was more like jfog. So a lot of companies, interesting places.

Love It, Love it. So, you know, one of the topics you said you wanted to cover was vibe coding. I guess people that know what is vibe coding is, a lot of people think it's a joke word or is this a legit thing? How would you explain what vibe coding is for someone that might not even know?

I will explain that this is something that, as opposed to when we used to call, you know, go to stack overflow and kind of check how to do things, you now have two things. One will be a chat that is actually helping you generate code and the one that actually generates the code. And if you think about it, it's kind of changed the way you define a developer. Who is a developer. If I just write, you know, a prompt saying, I want an application that I know color things in blue, is he a developer or maybe more a product manager trying to do requirements? So I think web quality in general really changed the way we think about developers of their day to day job. And I think the most important stuff is that if I'm using a lot of AI generated code, then the question will be who is the one that need to fix it? From vulnerability perspective, it creates vulnerabilities because it was created by people and hence most of the examples they see within the wild are with vulnerable code. And now the question will come, if I do scan it and I didn't write it, or at least I wrote a prompt to write it, can I fix it? Can I as a developer fix it? And I think this is really changing the way we should think about security for developers and how they really need to fix their code or actually help them secure the code.

You just mentioned something that's interesting. I was just going to say I've been using Cursor a lot and even small apps, and then I go, okay, check this for security issues. And it finds a bunch of security issues after it wrote me the code, which I always think it's weird. So you said one reason it's been trained, a lot of code and a lot of those code had known security issues. So how do you get around that? Is that always going to be an issue when someone's doing Vibe coding?

So I think the way to do it is to make sure it's being generated secure from the start. And trying to fix it after the fact is actually something. It's much more complicated. If you secure it, if you put it right, guardrails in place, if you put the right rules to be checked before that, then the code will be secure by generation and not just you know, after the fact. And also what I would, I would like to say is make me an application that doesn't have any critical vulnerabilities and not just create an application to do something. So instead of after the fact, change it to be before you actually create the code.

So it's, it actually could be a prompt issue then. So it's, you know, if you prompt it correctly from the start, then you're gonna bake in security practices. I guess.

Yes. But if you think about it, there are a lot of different things. For example, I may, you know, become kind of a loop generation of code. I try to do something which is not vulnerable. But what I'm asking is it actually has like vulnerabilities. So you have to be very careful in the way you define the prompt. Like what is. I would say it's the difference between perfect and secure enough. I think this is where the security companies come into place and say, we know what is secure application and we will help you bring in the rules to your AI generated code. If we just say, you know, write something with own vulnerabilities. I don't think I know any open source that doesn't have any vulnerabilities. Maybe something that was never used or never really in production.

Right, Right, absolutely. So could this give people a false sense of security then? If they said, oh, I told it from the beginning to write code without vulnerabilities, so I didn't have to worry about it at this point?

Yes, it may, and I wouldn't say it will completely remove all the tests that are being done later in the process. For example, for a pull request for periodic scanning for build, it will not kind of say, okay, we'll do it only on the IDE stuff and we'll do it only when developer write the code. But another thing that can happen and in reality it's something that actually, if you think about it, I will ask create an application or create a code which is not vulnerable. They will try to do that. But then what is actually vulnerable? I have like a lot of questions is the CVE is really vulnerable if I have it within my open source that I got may not be reachable. So I think it's kind of a balance between understanding which prompt you had to do and actually have a security that's in brain behind it to understand what need to be fixed. I wouldn't say that just a prompt and saying create something which is not vulnerable would be enough.

So I know before AI really came on the scene, it was more like you could have some tooling that as you're actually writing the code, it can like prompt you about certain suggestions how to make it more secure. But now that AI is writing it, you kind of lose out on that. So working for a security company, do you now, like, do you work with AI behind the scenes to make it more secure? Like, do you have any solutions to help with that type of scenario now?

Yes. So the idea is that first we have AI within our product for a lot of things, but we also change the way we think about AI generated code. And while AI generated code, when you go into the periodic stuff, like when you scan the branch, it looks the same, but the main issue is the developer experience, which is different because you cannot ask the developer to fix things. It didn't actually create it. And what we thinking of are kind of a level of security. One will be kind of a rules using our scanners today, then some enforcement going forward. There is also another attack surface coming from these ides with all this mcp. Think about the fact that an MCP can just ask you to delete all the files in your computer or that it can actually send all the code to an external source. So it's not just about secure coding, it's about the fact that you now have an autonomous agent that actually can do a lot of things on your environment.

Oh, that's interesting. Yeah. So almost your attack service has increased. How do you then know that? Like, how do you handle it? Because it seems like AI can then stay one step ahead of your secure practices. Or maybe that's not true, I don't know.

So I would say that both attackers and cybersecurity companies will use AI to make them and train about the future attacks. But for ide, at least for the vibe coding, it's a new attack service. Yes, people could have done mistakes in the past, but it's different than a malicious code coming. You know, I'm just writing a JIRA ticket and the JIRA ticket is saying, and then the geotech it creates the code and then geotech it can say just send everything to one, you know, a malicious website or something like that. So one of the option will be, you know, an agent or an EDR within your environment. Other option on hardware capabilities around the ide. So there are a lot of cool stuff that we are thinking about.

Nice. Do you have any real world studies or maybe how attacks may have increased with AI? Or how using tools to help you make your code more secure using AI has helped?

I think you see Kind of different things about attacks, different examples. But I don't see this as something that we can say it's widely spread because I think it's not yet being used as often. If you think about the big companies, they are still struggling with understanding how exactly to use cursor or cursor like components from security perspective. I mean.

So you know, OWASP has been around for a while. I'm always shocked how many developers and testers aren't aware of owasp. And you would think people would be like security to be top of mind. Is it has AI actually helped the security space, making people more top of mind? Is it still almost like an education process that people need to say, tell the management, hey, we need to watch out before we ship this?

I think it's more about making it easier. I'm not sure education will be enough because at the end if it's easier to write the code, it should be easier to write it securely. It shouldn't be something that you educate to write security about. You can see still put guardrails in place for all cases. And I will get to our SPM solution later on to explain. But generally speaking, I really think the product need to be easier and much more automatic. Rather than ask developer to write securely or ask them to scan the code, it need to be much simple. It needs to produce less false positive and be kind of adopted by simplicity rather than education.

That's absolutely true. I speak with a lot of security folks and they always say a lot of teams are worried that more security controls are going to slow them down and companies that say we need to ship quicker and faster. So how do you balance security with trying to deliver at velocity almost.

So this is a great question. Kind of brings me into the application security, the SPM part that we are doing. You know, one of the things that I think it's a perception that says we will not stop developers from creating application into production because we will kind of delay or slow the velocity of the business value coming to customers. But what ends up being is that problems or vulnerabilities get into production and then you need a developer to fix them. And then it's kind of starting a vicious cycle. I would say you have to figure out who is the one that actually created the code. It's not that easy to do that. Once you are in production. You need to figure out like which production issues actually map to a code issue. You want to understand, for example, how many production issues were created from the same code issue. Then usually you need to figure out, okay, this owner of the code probably already worked on something else. So you kind of bringing them back to solving things they did like few months ago or maybe a few weeks ago. And then they need to fix it. We need to build the software and then the software need to be deployed. So in the end you spend a lot of developer time just because we didn't want to stop them or kind of put the guardrails in place in the first phases of the software life cycle. And one of the things we focus on, our application security solution, beside other things I will get to this in a second, is that we will provide a smart guardrails. One of the things you usually kind of hear from customers, they're saying, I cannot stop any critical cve. This means that all my builds will be break or all my PRs will be break. I do want to do something which is much more smart in a way. Let's do it only for things that actually go to production. Let's do it for things that actually have an Internet access. Let's see if this, the worker that I'm deploying has an excessive permission, something that will give them more information whether this can really be exploited. And the entire say our perception around it or the motivation we did for the application security part was let's bring in all the information we have, whether this is the business criticality for the user, whether it's the code context, whether it's the cloud context and make them a super smart policies to be applied. So from one hand you will be secure while you get production and on the other hand it won't block your entire velocity or delay it to not be able to bring business values into your environment.

All right, so that's interesting. You talked about context. So you know, every business is different and you said, you know, depending what business you're in, something you may require more security than others. How do you learn about the context? Do you training on like the requirements? Do you like, do they almost have their own personal, not model, but personal kind of training. So it does have the context to say yeah, this is could be a security issue, but in this case don't have to worry about it.

So we have like two, three levels. One will be the code itself. We can know a lot about the code because it's scanning the entire code and you can understand whether this code was ever been shipped into production. The second will be about the cloud because we have such an extensive cloud security solution. We actually have, I would call it signals within the Cloud, whether it's on the network side of things, whether it's on the data side of things, identity workloads, we kind of see everything and we can actually understand how your production environment looks like. And then this context is we kind of bring it back to the application security and define it on the code and say we saw in production that this workload is really highly permissive so maybe we should be more careful about the things that go into this workload. And the last thing will be about the business criticality of things. Every organization have different ways they want to look at their applications. So this usually something that we need to do get from external sources. And the last thing, and this is a super exciting thing that I want to share. As you know, our Cortex cloud, Cortex cloud solution is built on our Cortex platform. And our platform also incorporates all our XAM and XDR and xo which is all the SOC and agent capabilities we have from the real attacks. So while you think about it, another context which is super important is what is actually happening in production. Can I say something about the attacks that I see on my production environment? Can I prioritize based on them? So having all these signals into our environment, everything on the same data lake and then bringing back to the developers and to the DevOps and say you need to fix it because these are the things and then also provide the remediation option and potentially in automatic ready mediation, this will make their life easier and then if it doesn't cost them anything, they would just apply it. And this is the reason we really believe that if you meet the developers at their systems, if you bring all the context into their environments, they will, I wouldn't say it's an education, it's more about just simplifying the process and make them part of it, part of the success, have like have a production which I wouldn't, wouldn't say zero issues, but a very small amount of issues. It's a victory for everyone because nobody will be, you know, get up at night for security issues. That was just found on production, which is obviously much more urgent than anything will be found in code.

Yeah, absolutely. And I guess maybe this isn't right either. But because you're using a product like yours, a real product that knows about other attacks that are happening, maybe they're newer attacks that are just started happening in other companies and you know, someone, a security agent maybe needs to stay on top of, oh, I better check my code for that. Does this also give them insights? Hey, here's a new vulnerability we Found going out there. And by the way, you have it. So you might get on. And now is that possible?

Yeah. So the models we are training for attacks are always in continuous updating. So we have this information from there, I would say the amount of data we have, not just from a specific user. And then the idea is really to get them back to the developer and say we saw this attack or we saw something new that you see and we want you to fix it. And again not fix it, create the code correctly. I will give you one example which is very common. Developers like to, I would say, choose an open source components that do what they need. And these components are not necessarily something you can trust. If you think about an agent that just tried to generate a code based on oss, he will probably use the one he saw that is being used a lot. But it doesn't mean that it's a secure one. If this will be flagged out within the ide, then no issue, the code will not be created. You will create a code with a different component and then you will continue to do your regular coding. If this will not be flagged within the IDE, then you're already stuck with an APIs and an OSS component that you are already chosen. And then you have to figure out how you patch it or maybe upgrade it or something like that. So early prevention, which is something we really believe in, is important for everyone just not to spend time on things that you could have solved if you planned correctly.

Nice. So we've been talking about as developers and security, but I know security. There's a lot of trained professionals just in security alone. Is there such thing as like Vibe security where someone just now can use plain English to do security prompting rather than have to go through all the certifications and everything that a lot of them require to have.

So think about like simple tasks. I'm security practitioner, I can be knapsack security practitioners. And I need to know I'm responsible for an application and I want to understand what are the top items for me. And this is something that AI can help a lot just, you know, by going with natural language and ask what are my top issues to fix. The second thing will be what are the things I need to do to be able to prevent first? And it's not just fixing things, but also making sure I'm kind of lowering the funnel and making sure not no more issues are coming into the funnel. So I'm a great believer in natural language and using AI for I would say a lot of tasks I also believe it will change the way we think about products and how products are being built.

So you mentioned production a lot. Does that highlight areas like, you know, this isn't necessarily a vulnerability in security, but we noticed this module is being used a lot so you may want to add extra security measures around it.

So one of the things that we do is that we have a scanning of the models and we have full inventory of the models we have in production. And we also have these different signals within your pipelines and within other places within the environment to figure out if there are models that were deployed, although you don't have the permission to do that. So yes, AI is super helpful, but it's also something you need to figure out if you understand exactly what is being deployed, if you understand exactly what is being used. So we're kind of trying to cover everything from code to the runtime and to the SoC.

Gotcha. You did mention God Rails earlier, like other. Can you like list out some maybe guardrails that could be put into place to help make more secure code with AI?

Yeah, so I gave an example of a bad guardrail. Just block any critical CV that you are trying to submit or every critical and high a good guard. We say please make sure that these CVs are really reachable in code. They are on business critical application and being deployed to production. And again I would say that if I was naive, I would say that everything, every vulnerability within the code will be fixed. But we know that companies have their own targets, goals, they have to bring business values into the environment. And we are here to help them secure the things that are important. I wish they had time to do everything, but they're more realistic than this and hence we have to figure out how to help them really fix the things that matter.

So over time do you foresee security no longer being an issue because AI will then be handling all the security for you? I mean, I don't know, that's probably outrageous, but any thoughts on the future of it?

I would like to say it will be simpler. I'm not sure it will not be an issue, but I think in my vision it needs to be much more simple, much more automatic in the sense of how to remediate things. And it has to be incorporated within the processes in a very smooth way. It cannot be something that will be done after the fact, but it has to be as part of your day to day job. So whether it still can be a burden of, you know, trying to fix, trying to scan a code. Yes, code Will be probably easier to be created without any rules. But you know, we are living in a place with rural, so I would assume it will apply to code as well.

Nice. So you have a lot of products in the Palo Alto network. So I'm just curious to know if a company's just starting their security journey and they may feel overwhelmed. Like is there stages you recommend going implementing this? So that's they're not like throw it all at the wall and all of a sudden like they, they have to have all these false positives they need to work through. Like how does that work?

Yes, so definitely especially on application security because usually companies have a lot of different problems. There are some things like we added, which is super cool, we call it kind of stop the bleeding. You only fix things that are new, so at least you understand you have other issues and you have a huge backlog or technical depth that you have to fix. But the first thing you are doing, you are trying to block all the new stuff that are coming in. So trying to do it, you know, in a more gradual kind of more in steps or in phases rather than just say okay, block everything. This is one example. Another example is putting some goals to a group saying you have to fix like 15% of your backlog by this quarter. Another thing will be about the different.

Scanners secrets. We have the option for example to validate secret so we can understand whether a secret is actually being used. And this kind of give another level of prioritization. So the idea is really to figure out how we can help them prioritize the work and understand what are the remediation they need to do. And the second thing, and this is the most important part, is not to create an issue in the first place. And I think my example on choosing your own open source component is a very important in sense of not getting into a place you have to fix a lot of critical vulnerabilities.

Great, great. So you know, have you heard a lot of. Do you hear a lot of things that maybe aren't true about AI generated code from a security perspective? You're like, oh, why does everyone think this? If they only didn't think this, you know, it'd be so much better.

I think AI is here. It's not a question for us, it's here. I think all of us are using.

Like chats and we do a lot of things with using either lamps or AI or agent in general. I think this is the new way things should be done. I think that if code can be generated by AI, then we just need to join this one and not say it's vulnerable or it's not going to be in the same quality of things. I do believe it's something that's already here. Yes, there are some kind of concerns about the type of code and we do see that some AI models tend to do some, I would say things that are not like deleting code from production or doing things that are just not as we want this to be behaving, but it's here. So we just need to make sure we do it securely. I don't think we need to fight the new stuff coming in, but rather embrace it and make sure we kind of use it for our advantage.

So that's a good point. I know a lot of developers and testers that say it's just hype, it's gonna be an AI bubble, this is all gonna go away. But I know that for security a lot of people are using AI and they are making it easier to do hacks. I think so. I think so. That type of thing. It sounds like it would leave people open for more vulnerabilities if they don't start embracing AI and realize it is here. I guess, yeah.

But we can use AI to protect.

As well, to protect us. Right, right, right.

So it's kind of a race for protection and for an exploitation in the same way. I believe the protection stuff will be winning, kind of the security will win here.

Awesome. All right, before we go, is there one piece of actual advice you can give to someone to help them with their DevSecop efforts or security efforts? What's the best way to find contact you or learn more about Palo Alto networks?

So of course in our site for Palo Alto, look for spm, super cool announcement and product that we are having. We kind of go through our entire ecosystem as a platform making sure that developers has their own way of using the platform and also application security. You know, one of the biggest challenge will be trying to kind of bridge the gap between application security and developers. And the second thing I would say application security is a journey. It's not in one time. You have to kind of set your KPIs and see how you are progressing with them. And we are here to help and kind of, you know, help with explaining what is top important and then what are the things that can be fixed later on.

Absolutely. And we'll have links to all this awesomeness down below. Thanks again for your automation awesomeness. The links to everything of value we covered in this episode head on over to testguild.com a567 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good.

Cheers.

Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we.

Can achieve O the Tescill Automation Testing Podcast with lutes and lyres, the bards began their song. A tune of knowledge, a melody of code through the air it spread like wildfire through the land, Guiding testers, showing us secrets to behold.