Vibe Coding vs. the CISO

Thanks for tuning in to Simplifying Cyber. I'm Aaron Pritz, and Cody Rivers is not here today. He's visiting the mouse and emptying his wallet to the Disney lords for the week in PTO and much deserved. I pulled in Bronwyn here, who's on screen and listening to you online, for a little special challenge. I ended up vibecoding an application that does a pretty good swath of one of our full services and very strategically used some intellectual property that was safe, ours, not any of our customers or whatnot, and built out this entire application. And I'm about to drop into a meeting with Chris Addicks, our CISO, and a managing director, Todd Wilkinson, who leads a lot of our technical services and a lot of our AI projects, and Michael Milroy, who was the first Reveal Risk vibe coder, and I caught the FOMO seeing what he was doing and that one thing led to another and that led to my little special project here. So the goal, they don't know what is about to happen. I've pounded their calendar in one of their standing one-on-ones between two of them and added in two more people. They saw the Riverside recording link and now they're fully freaking out, but we'll see how we manage that. And I'm going to do a quick demo of the application that we built, or I built really, Claude, Claude Code built, uh, and I, I'll take all the credit, but I actually tried that. I said, what What code, what language did we develop this in? And Claude Code was like, you, to be clear, you didn't develop any of this code. I did it all, but here's the languages that I used. It was very, it was very pompous. So anyway, that aside, we're going to drop into this meeting. It's going to be maybe a little awkward at first. We're going to talk about this and then we're going to shift the conversation to how CISOs, cyber leaders, IT leaders, could and should be handling this, whether they want to acknowledge that the stuff is happening or they blocked it and they think that it's not happening, it is, or they're really leaning in to try to figure out how to enable innovation and put the right controls in place to make sure that the progress doesn't get chutes and ladders back to the beginning of the board. Without further ado, here we go. All right, so this is a little bit of a simulation. You guys, so Chris, you are in the simulation as well as are in for real life, the CISO of our company. Michael, you're doing a lot of tech development, have played, played with AI. Todd, you're AI product advisor and you're getting a lot of client questions about AI, which all of that is absolutely true. So what I want to do is, um, Todd and I were in a conversation with a customer last week and he was talking about the CEO is just like hardcore in the vibe coding and he's having to give data access to them to kind of try to contain it, try to enable him to do what he's doing. What he's doing is really cool, but he's trying to kind of figure out how do you enable the innovation but also control the, just the out of control, like what if he uses client data? What if he pinholes access to things that he shouldn't. So it's a new landscape, new control stuff. So what I'm going to do here is I emulated what he's doing in our own environment, and I'm going to show— I'm going to do a quick demo of what I did because it's interesting, it's cool. But I also want to have a little bit of an impromptu discussion on what our client should be doing to not necessarily prevent this, but get our arms around the fact that AI is leapfrogging on a weekly or monthly basis and executives and employees are all out there trying stuff and cyber in many cases is not at the table.

All right.

So, and I'm going to be a little provocative here for, I did do this in a very controlled fashion, but I'm going to, I'm going to, I'm going to, I'm going to overplay what I did just for effect. So I took all of our third-party risk management intellectual property and I uploaded it into Claude code. From Claude code, I— third-party risk is a tough topic. There's GRC tools that we've tried. It had really painful. So within the confines of a few hours over the weekend and then a couple nights this weekend, I iterated probably total cumulative of 10 hours to take all of our IP and develop our entire process in a fully functioning SaaS app that's running locally right now, but could easily be replatformed and deployed in the cloud or on Azure or whatnot. So just, just to kind of talk about the tool for a little bit and kind of show how it aligns to our process. Right now we're looking at the analyst queue. Kind of showing the actions. We've got the overall platform, some really cool metrics of high-tier open findings, average daily cycle time, all coming from Michael's metrics packages that he's put in place before. Some really cool summary, like risk by domains, you know, general, obviously that's bundling things together, but data protection, access control. From a risk summary standpoint, I can drill a little bit more detail. I can see that we're only remediating 12% of our findings. And I can, you know, see more specific things on risk domain and get into details with reveal risk recommendations, which we could tune on the bigger themes. So we could actually action enterprise-level decisions based upon the themes we're getting out of third-party. With our process, obviously tiering things by t-shirt size is important. And also we were able to add AI risk analysis on the vendors itself. So you could kind of get an early look before you even get into the assessment itself, and then confirm from an analyst standpoint, you know, do we believe that that would change, you know, that individual risk and override, describe, things like that. And obviously we can, we can see the list of vendors. This is all hypothetical data. You can see here we have the Addix Art Bar and Grill. And I only chose art because there's an Addix Art out there. I wanted a real URL that I could scan. Yeah, exactly. And then we can see we've sent a, you know, an assessment to them, a rapid triage. So we could get back that back and then be able to have analyst analysis on the backend as well as the risk scores and then specific contacts that would come up. This is where I onboarded that vendor before this call. And then questionnaires, obviously we can load in. I loaded in a few samples of questionnaires we have from our frameworks library, you know, the pharma assessment, the rapid triage, and then this was an evidence checklist. Interestingly enough, I also took the Natera redacted data to do the additional enhanced due diligence to put additional actions that we could put in place for specific vendors. And then monitoring-wise, obviously you can see some alerts. It was able to, let me go down here. Settings. Oh yeah, DDQs and answers library and a full AI capability to match questions to answers banks, which I think we've struggled with, with some of the external tools, but at least some basic capability to log an incoming DDQ, drop in the questionnaire, and then have it match to the, the current answer set that we have. And then obviously risk register. Been able to roll all those findings up, get some specific reports that you can push to Excel or I guess PDF, PowerPoint, and then also custom framework building. If we aren't going to use a standard assessment, we can create a custom framework itself to be able to use that for that. Anyway, that's the quick demo, not being an SE, just to kind of say, you know, I haven't touched code in 20-some years. It was able to fully build out the database schema, I think over 10,000 lines of code to support it and do it all based upon business requirements and vibe coding and specific accelerators that we could load in. So pause there. And by the way, I very carefully chose specific accelerators that were out of date. And not that recent and/or not something that every other company would do. But let's assume I'm not a cybersecurity CEO and I don't necessarily understand the risks of putting this stuff in there. Let's assume I dumped our entire SharePoint library in. So let's start with Chris first. As the CISO, what controls would you want to put in place if you knew I and 3 other people within their jobs, we're going to be doing this without a lot of IT. Our current acceptable use policy obviously doesn't contemplate something this advanced. What's your thoughts? And really, our clients are facing this as well. Like, how do we help them get in front of kind of some of this freight train that's moving, whether we're on the train or not?

Okay. All right. So, so here, so let me, let me take a second. I, and this, this, this is aimed at us in general, everyone using AI, like, as cyber practitioners. I'm not quite sure why this is so confusing for everyone to kind of get their arms around when we think about, we've done this over and over again with technologies that come, have come up, right? Internet, cloud. You name it over and over again. And this one is obviously a little different that it's very fast-paced. It's moving way faster than we can all imagine and leaps and bounds, like you said, but the same concepts apply that we've done for 30 years. It's make sure that we're taking the things that we know, the things that we've done, the risks that we've managed and apply them to a new technology. So how to do this? The answer is yes, CEO, go to town, have fun. However, What would we do on any other platform? We'd say, make sure you have your third-party risk assessment done. Make sure the data's being properly controlled. Make sure the vulnerabilities are being managed. Make sure you're managing access in an environment, right? And I know these aren't a CEO waking up in the morning going, I'm gonna go do this, but that's happened. That's happened with Dropbox and Box, the whole file sharing thing we went through many years ago that everybody was using. It took us a while to get our arms around, but ultimately, what do we do? We applied the same concepts in a different way, more advanced, more complicated as we, as we continue to advance technologically. But I think same thing here, guys, like, how are we going to do that? The answer is yes, go and do it with these controls and applying the controls that we've done over and over again.

And how do you feel on blocking, right? Because like when I, when I was on the corporate side a decade ago, cloud storage came out, we had some insider threat, the legal reaction was we'll block, block it all, we'll people pinhole access to the stuff that are approved tools. But then we saw the problem squishing around. People were going to home personal devices to get their work done in other ways or use their preferred, you know, cloud. Yeah.

There has to be some control there. We can't go, well, use what you want, right? Because there's a couple of things. You're managing cyber risk, but then you're also managing the optics of risk or the culture of risk, right? Saying, if a company takes a stance, go use anything you want, have fun. Who's, where's that, where's the responsibility for that risk lie? The company, right? If you're sitting there going, well, here's our approved, you can't use these things, and something happens, that positions the company very differently in that risk conversation when opposing counsel's going, why did this happen? Or how'd you let this happen? Well, we were like, it was a free-for-all, right? No, that person violated the policy and the technical controls and ended up doing X. Different conversation, right? So I do think I do think there is the need for reasonable blocking. I do. However, like anything, you can't say don't use Box or Dropbox and not provide the employee base with something else, because that's silliness, because that's exactly what you said. People are just going to go around and figure it out. I'm going to use Mega Upload to share with my vendor, right? So ultimately, looking at this, it's you can't do this, but we're offering this, and it's just as viable as the other one. And then over time, as we continue to advance in AI world, opening more and more capabilities and abilities that we get our ability to manage risk in the technology.

Yeah. Michael, you've been a proponent of, of vibe coding and product development and alignment to the process stuff. Obviously you, similar to me, practitioner in cyber, know some of the precautions and dangers to avoid. As you, as you were leaning into it and approaching it, like, As a, as a, as a leader, you know, in the company, like what did you think about, how did you self-regulate maybe some of the things you could have done but didn't want to, didn't think it was the right thing to do? What was your, what was your thought pattern as you were exploring into this stuff?

Yeah, so like when I did the, like with Assessor, kind of the, the first round with it, um, that was very, very heavily on taking, taking what we have and building from scratch to, to kind of mirror some of it and then expand. So I wasn't taking anything of ours and putting it into another tool. I was pulling it up and saying, I like the outline. I like the structure. I want to use this as a foundation, but I'm building it on my own. To, you know, so it takes a few extra hours up front to kind of build that, but then all the additional functionality. So kind of some of the more recent side projects, taking, sanitizing all of those things like on my own. And then, then I'm comfortable running those, those documents and things through, through the platforms and stuff like that. But I mean, some of the key things I looked at is like, from a security perspective, I know the one you just showed, you're running it locally. However, if somebody got ahold of your device locally, is now local to them also. So there's one issue. The other thing is if, if you were to take the next step and say, well, I'm already running it locally on my own, I'm going to share it with the team. And you just go and post it somewhere else? Are you posting in a hosting place that is safe, secure, vetted? Have you run any like security testing against it? Not like full-blown pen tests, but have you done any security checks to see kind of what the access control and stuff like that is? So those are things where I've kept it very much internal, sanitized at my own, or again, looked at it, built it from scratch, and then expanded upon with minimizing what I share with some of those tools.

Yep. For people in marketing or ops outside of IT, outside of cyber that are jumping into this, how many, without guidance, Chris, to your point, without any kind of structure governance, happy path, here's the job aid to do it the right way, how many people outside of IT and cyber do we think are thinking like Michael is?

Not many. And I actually just saw 3 posts on LinkedIn within yesterday alone. 2 of them were legal firms where junior and senior attorneys are just dumping everything, client documents, client meeting transcripts, everything, no sanitizing because no one told them that they couldn't. And to them, I'm saving 40 hours a week because I can dump it in there and I've not been told I can't do it. So it's making my job easier. So literally 3 cases that I saw, I read them yesterday, but they all happened within the last 2 weeks. So yeah, I would say not many.

The thing about it is like a lot of us want to enable the technology. There are CISOs that don't, but trying to hold it back or say, hey, we're not going to get into this yet, it will find a way. The Jurassic Park quote, like life will find a way. And it's the same thing here. People are going to find a way to innovate. If you're telling them they can't, they will find that way. Todd, from an identity standpoint, we were chatting in the office earlier, like this kind of expands the aperture of identity and managing, you know, different forms of bots and agents and things like that. Like, how do you think companies are faring in that early battle?

I'm not sure the identity teams are truly ready for this. And if you, if you work through how some of these code, you know, these, these vibe coding tools are working and how they're setting it up, one of the first things you need is access to that data. And it's easy to start with, give me the documents and just let me throw it in here and let me read it in. That's one of the first places you start. But the second place you start is, let me connect to my database, let me bypass the reporting tools. And IT teams over the years have been very good at going, I'm going to control sensitive credential, I'm going to put it someplace secure, and then I'm going to present that report to you that filters that data to you in the right way, in the right place, and I keep that sensitive credential safe. Now this has pivoted to where you're going to have a lot of people that are not developers, who are not practiced IT individuals going, I need that sensitive credential, I'm going to put it on my laptop. It isn't going to be encrypted. It's going to be in a plaintext file. And worse, that credential likely does not have MFA. It is probably completely open. It's going to bypass those rules. And that is going to explode and increase. And you're going to have info stealers that are going to grab those. And those accounts, by the way, are also monitored less. It's the nature of security that that's why you try to put those things and hide them away. So you may see security teams come back and say, okay, you can do Vivecoding, but we've got to create an environment for you to do it securely. And that's that doesn't include your laptop. That might be a place to start because some of the new technologies and methods to protect those identities are a little bit different than what most are used to. They're new products, and they're not geared to your average person. They're not geared to your finance person going, just give me access to the SAP data and let me start running reports. I'm going to bypass Excel. I'm going to bypass SAP. I'm going to create my own reports here on the own. I think that's a challenge area that we're going to have to work through. And I think security teams are going to have to not only lean into new tools, they're going to have to lean into new audiences that they typically haven't communicated to. They're going to have to talk about development practices to non-developers, and that's going to be an interesting convergence to happen. I mean, can you imagine talking about a credential vault? Not a password vault, but a credential or an API vault to somebody in finance. Not to diminish finance, but I'm going to guess using developer tools is not their, not, not in their forte of practice.

Yeah. Michael has his hand up and I'll, as I, as we pass the mic to him, I'll say, when I started this project in Cloud Code, it first asked me like 3 times for a GitHub account, which I had a corporate one, which wasn't directly within GitHub. So it was really pushing me to set up what would have been a personal GitHub account, which would have hosted all the code, whether I set it to be public or private or whatnot. Would have been up to my knowledge of, you know, whether I should do that or not. So to your point of like, once you go online and it, you know, it'll help you get online real quick. It'll help you set up third-party app files. It's really helpful with that. Like probably saved me hours, but I think it's a slippery slope without the right guidance. Over to you, Michael.

Yeah.

So that was one of the two things I was going to say was, was the GitHub connection for sure. Especially for people who aren't coders, probably don't even know what GitHub is or how to handle it, like how to even think about security within it. So that, that's definitely one and it is extremely helpful and it'll do 98% of it for you and create walkthrough documents for the other 2%, right? The other thing is Aaron, when you built this app, when you went to go use it, even for testing or demo, did you, did you create a login page to where you had to log in to test what you were doing?

I did, but, but that's because I knew that I wanted security in it. If I, just wanted to do straight up development and not be burdened with that step, that would be an issue.

Right, exactly. So how many people, even inside security, to be honest, I know security is, doctors are the worst patients, right? So when people are creating these tools, I'm creating this tool to save me time. Why would I cause myself more effort by making myself log into a tool that I'm making to save time? So it's just, it's another one of those easy things of maybe it starts out, well, I'm the only one using it, so I don't need a login. But then, oh, hey Aaron, I want you to check this out. Uh, no, there's no login. It's just us. It's fine. And then it does start to scale. So then it becomes hard to add, add some of those security things once people have been using it.

Yep. Well, one last question for Charismatically Curious and CISO Chris.

Wow.

That was a four, four. For wordplay alliteration. Are you going to now block Cloud Cloak Code and kill my fun? Or what?

I did it while we were talking.

And what are the next steps to protect our own selves or protect me from my mayhem? I'll say mayhem.

Good question. I don't know if I could protect you from that, Aaron. Um, let's see here. Uh, no, I think, I think it's a little, we're in a little bit of a different situation. You understand the risk. So I think if you didn't, And, uh, we were a similar size company. I would try to listen to what you're trying to do and get our arms around it to put the controls in on the fly while you're doing your thing. Or say, hey Aaron, we know you're on a full head of steam here, but pump the brakes for a second. Let's organize some stuff, make sure we're at least doing some of the basics here for you. And then maybe get some agreements in place with Claude, get the enterprise you know, kind of agreement in place and at least get something there. So we're not just using the publicly available version and there's no security controls in place. So something, not everything.

And I did get coaching before I went to set up like a POC environment and turn off the training on our data and things like that. But there's more steps, you know, as we would progress that we will, and we will need, we will need in actuality. As well as we would advise for our clients as their CEOs are probably doing this and not just CEOs, but leaders and employees of all levels are, you know, experimenting, right? It's, it's fun. One of our CIO clients said he spent 6 hours on a holiday weekend trying to set up an environment to help his CEO, not because he had to, but he was like, I'm actually having fun with this. So that when there's FOMO or fun, actually, Michael, the only reason I started vibe coding because I saw that you were doing it and I'm like, well, shit, I gotta get caught up here. This seems like something I'm missing out on.

I think one of the other things, so the GitHub thing is an easy way that security and IT can start to bring other people into, kind of into the fold and encourage it, right? Because this is, there's a lot of things that finance would come up with that we would never consider being a need or something like that. So I think that there, it would be a miss if IT and security just shuts everything down. Whereas if it's, hey, we already have an enterprise GitHub or, you know, a company repo that we use, bring them in. You can segment them off to a separate section so they can't access other production code, but you can bring them in, hook them up, say, hey, absolutely, like, you know, dump your code in here. It's at least more secure. Start with that. Encourage it, but then at the same time, you're also encouraging and enforcing some of those basic security controls.

Well, life will find a way whether we are protecting it or not. Had another CISO that we work with that, you know, I was sharing last at a breakfast coffee meeting across CISOs that I was playing with Cloud Code and he was like, oh yeah, I am too. And I said, how are you doing it? He was like, well, I've got an old Mac MacBook, you know, company device and I'm doing it all there. You know, so again, he was thinking about ways to not do it on his core laptop, but it was a company device. But anyway, I think it's more important for us as practitioners. If we don't know what these tools are, if we don't know what the potential is, we don't know what we're protecting.

I think there's a little bit about kind of front-ending the conversation with business need and what should be done versus kind of open playing field, right? Because I think that's one of the first things you do to manage risk. If you say, well, everyone at every level of the company can go and tinker. Okay. Well, that's very broad, but number one, that's introducing more cyber risks than necessary because everyone's doing something. And then on the other hand, potentially wasted time for a company, right? Maybe there is some front-ending of this with a business process to develop business cases to allow people to focus on business-supporting initiatives or code to rather than just kind of, hey, I have an idea one morning. Now, When you get to the leadership team level, CEO, COO going, I'm gonna go try something, that's different. That's a white glove VIP experience. They're gonna go and do that and you support them. But having the entire accounting team going, huh, I got an idea, and go, go to town typing. Does that even support the mission? Does that support the business? Right? So wasted time. And that's, that's the first place you could probably reduce some of the cyber risk rather than having 30 people go tinker on a weekend, maybe Maybe it's 5 people tinkering because it's an approved mission supporting idea. And they're being, they had that scaffolding around them to go, yeah, go and give that a try and come back. Let us see how it works. So just some thoughts there, kind of riffing on what Michael said.

Thanks, Michael and Todd, for letting me steal the 10 minutes that turned into a full 30 minutes of your one-on-one. But I think this was a fun conversation. I think this is a, uh, special edition of Simplifying Cyber.

Yeah.

Cool. All right.

See you guys. Thanks.