Above the Stack Ep 14: AI Security Beyond Policy: How MSPs Can Put Real Controls Around AI Use

Communications arrays are aligned.

Nobody cares about your stack, bro. Well, well, well. I hope y'all are doing great today. We had a little bit of a snafu, Nick Ross. I think we accidentally gave out the link to Or somehow the link for the actual Restream got shared out to everyone, so people are trying to join here. If you're trying to join and cannot get in, or you're already in, sorry about that. This is meant just to be a livestream, so we're just—

you know what we're gonna do, Nick?

We're just gonna YOLO it. How does that sound?

It sounds good. Well, we're gonna run when they come in.

Yeah, sometimes mistakes like this happen. It's not a big deal. We're just gonna roll with it. But I still see room is full, all these people trying to to log in. So sorry about that. If you are in the live stream here in Restream with us, we gave you the wrong link. If you head over to LinkedIn or YouTube on either Cloud Capsule's page or Impact's page, you'll see this and all will be right in the world. You'll see it all live stream there, and that's the best place to be. So sorry about that. Those things happen. Nick, how you been, man?

Been good. I actually took a little vacation last week, went to Alaska. So it's pretty fun. Got out in nature, got away from the tech for a little bit, which is always nice. Yeah, recharge. How about you?

Uh, I did not go to Alaska. Uh, just been busy, uh, with all the things. But we did have a good 3-day weekend, went over some friend's house and just kind of chilled. So did you see any bears or, or whales or anything else epic?

No whales, but I did see grizzly bear and black bear, so that was cool.

Nice.

Otters, sea lion, and all kinds of life out there. So it was pretty awesome.

Dude, that's fantastic. Yeah, we went to Montana earlier this, um, summer when it was still like cold and wintry up there and saw a bunch of bears, and, uh, that was super dope. Uh, so yeah, they're pretty awesome to see in real life. Well, this is gonna be a fun discussion. Um, we've kind of been going through this journey. For those of you that are, uh, joining with us and been with us for a while, or if it's your first time with us, welcome. Welcome to Above the Stack. This is a really fun livestream that Nick and I get the privilege of walking through. What it's really about is helping us move in our journey as MSPs, sticking less to technology, even though technology is integrated into everything we are and who we are, but really understanding where our clients demand and expect of us is to get to the strategic layer. So we're getting to what we always call focusing on business outcomes. And I know that a lot of times we say that word, those words flippantly and no one defines what they mean. But ultimately, how do we move the business forward? And not meaning our MSP, but the client. And so we've been going through this really great journey. If you've missed the other episodes, you've got to go back and watch them because they really are prerequisites to what we're going to be covering and what we're continuing to cover. We've walked through this journey of the fact that your clients are already using AI today and you're just not in the middle of that conversation.

Or maybe you are and you're not where you need to be.

And all the way into the sales efforts and the marketing efforts and how clients think about it. And today I'm really excited because we're gonna be talking about AI security beyond the policy, and we're bringing somebody in who I think is not only a fantastic and awesome guy, but also somebody that has a lot to say on this. And so if we have not had a chance to meet your 2 guests or hosts, myself and Nick Ross, but I'm gonna go ahead and bring in without any further ado our guest, the one and the the only, uh, Mikey Pruitt. Let me get Mikey in here if I can click the right buttons. Uh, Mikey, how you doing today?

Oh, doing fabulous. I was also not in Alaska though, so kind of, kind of disappointed about that part, right?

Yeah, we— next time we got to make this happen in Alaska.

Yeah, just think of the cooling bills, how much lower they could be if the data centers were in Alaska.

Exactly right.

And we can just, you know, suck all the glaciers up for the water.

I mean, it's perfect.

I do still see our room is jammed full. If you're sitting in the Restream with us, just to let everyone know, I think we accidentally gave out the wrong link. The best place to watch this is going over to our YouTube or LinkedIn pages, and you'll see it live streamed out. But I see people still trying to join in because I think we gave the wrong link. So sorry about that for all the folks that are watching and trying to get in. It's like the room is full. I don't know what's going on. Just head over to the live stream and you'll see it. Mickey, I'd love for you to get us on the playing field a little bit on who the heck you are and what you do over at the awesome DNSFilter.

Sure. So I run AI Labs at DNSFilter. I guess it's a really relatively new department. And my job is really to make our team of about 150 appear like, you know, a team of 1,000 in the work that we can output. I spend my days connecting all of our systems like CRM, support ticketing systems, HR systems to our AI tooling and the scars and tribulations that come along with all of that.

Yeah, man, I love that. And how did you get into a role like that?

Well, that I guess is interesting. I have been dabbling with AI stuff since 2023 or so when ChatGPT came out. I was very prolific in copy-pasting code from ChatGPT windows trying to actually build something. And that kind of devolved into, you know, more programmatic building of things. And our CEO, Ken Carnesi, was like, hey, we need someone to head up this AI department, and I would like for you to do it. And I was like, thank you, because that's exactly what I want to do. So I have this knack for, at DNSFilter, doing the next thing that is most attractive to my brain. So it's always, always worked out really well.

That's fantastic. Yeah, I love that. I mean, that's the best way to go, right? We're all kind of figuring this out, right, Nick?

Yeah, I think it's awesome you guys have kind of like an incubator department. I think a lot of people, if they don't already, should have that, right? As far as like kind of an experimentation, but also it's kind of like its own hands-on learning center, right? For things that are moving, things that are changing. Obviously it's growing at a rapid pace here, but just having that as kind of An incubator is a pretty cool concept.

Yeah, we get to do some of the fun stuff for sure, like testing out new models and playing with local stuff and testing out new things. But a lot of it is kind of a support role, enabling the rest of our team, giving them the power-up capabilities that we're all kind of reading about and making those a reality. And I think that's actually where the MSP sits in all of this AI world.

Yeah, it does. And that's exactly what we're gonna get into talking through all this. And I think what's gonna make this a fun conversation is like, Mickey's— Mickey, my brain is not—

I've been called worse. I've been called worse.

In the pre-show, I kept saying Mickey, and I know Mikey well. It's just my brain is in Mickey mode apparently. What I love, Mikey, is that you've already been doing this with DNSFilter. So I think what we're gonna start to do as we jump into this a bit deeper is lean on like what you've done, what you've experienced, what you, what you would encourage others to be thinking about when it comes to this role. And so as we kind of jump in, one of the things I just wanna let everyone know, and we'll come back to this multiple times, but we do have a giveaway today. And I wanna say, Nick, thanks for being the awesome guy you are. This one actually comes from Cloud Capsule. Nick and team over there have already written a guide just on securing cloud, and that's gonna be our giveaway today if you guys are interested in grabbing that. Nick, you wanna speak a little bit more about what this is and what its purpose is?

Yeah, it's really more of a North Star hardening guide for Claude. So just kind of getting started with the basics of, hey, let's get a managed account and get some governance around the usage of Claude within an organization. But then goes through some of the things that you might want to turn off that are on by default. All as a kind of a similar standard for me, like I'm a big Microsoft guy, obviously. So similar standard in the sense that it's built for productivity, not necessarily for security. So a lot of things that you might want to go through that will disable some more of the risky features in Claude, but also get users into a, you know, compliant workflow that you can somewhat monitor, but also includes kind of a feature matrix as well too, just based off the plans that you guys might be on. So more of a how-to guide, but also something that you guys can take a look at to begin to harden, you know, your instance, whether that be getting that governance around that or starting to put things into place like single sign-on with an identity provider like Okta. Microsoft as an example, and controlling access into the tooling and some of those things, as well, too. So definitely a lot to unpack in that guide, but certainly a lot of places where you could do, like, 3 things and get a lot of security benefit, right, in any organization.

That's stellar. So we're going to leave that up, and what we'll do is we'll have just this little QR code. If you guys ever— when you're ready to grab that, it'll be just kind of right up there somewhere at the top of the screen, and you can grab that anytime you need. So let's kind of jump into this a little bit. Mikey, one of the things I think I want to start with is we've, I think, well established the fact that MSPs are encountering AI-based conversations every single day, that most clients are already using AI in some form or fashion, even if it's basic, just chatbot functionality. They're heading over to Copilot and asking a question, going to chat and asking it a question, right? But I think if we're starting there, the fact that a client has already got AI in some type of usage, but they're probably not building agents, they're probably not thinking about how do I do this correctly, they've probably never given any thought to data governance, to all of these kinds of things. Help us kind of understand how do we go from there? Like, help us in the beginning of this discussion and journey. How do we start there and start to deepen our conversations with our clients in order to get to a stage where we can start to deploy AI correctly and securely? Help us kind of get on the playing field here of where you would start in those kind of conversations.

Yeah, I mean, you're exactly right that nobody rolled out AI. It just kind of showed up, very similar to bring your own device back in the day. And now we're all in this world where we have to deal with it. And I think very much like that is starting with inventory. You know, where is the traffic going? And I don't want to be, you know, give the DNS filter pitch, but you need to have some observability into what tools are being used. And DNS does see everything first and is a great layer to be able to visualize what is actually happening. So every, you know, Claude, ChatGPT, all of those accounts, even some local stuff, does use the DNS system so that you can figure out what are people doing. And that's beneficial for a few reasons. And I want to say this really, really upfront, is that your job in the beginning is not to just block. things. Like, you know, you have to inventory first before you clamp down with some type of controls. So, you know, we've got connections to all of our stuff, and then you're basically going to implement those controls after that.

Mm-hmm.

But that inventory layer is where you can see maybe where people are spending time, where they're getting value from, things you may want to officially kind of bless, I guess, is a good way to say it. I mean, we had this dance at DNSFilter. You know, it's easy for us because we can see everything, but we were concerned with, you know, offshore model usage, MCP servers that were like unsanctioned. So the first thing we did is kind of create a policy document. Now, we all know that No one's going to probably even read that document or follow it, adhere to it 100%. So that— but that is a good, like, guiding star. And you can even use that document when you're— as, like, context into your agent space so that it knows what you are expecting the outcomes to be. But that inventory piece, like, where are people going, what are they doing, and what data is perhaps leaving the environment? Is step one.

Yeah, I think that's huge. You know, we recently went through our SOC 2 Type 2 audit renewal here. And I had them explicitly, while they're not like explicitly on SOC yet, I had them add AI controls, because I thought it'd be a good exercise for us to go through as well, just with our own inventory and usage. And, you know, we wrote a, you know, policy document behind it. And it's certainly hard to write, right? You know, even for a smaller company like ours, you know, in the sense of employee count, um, you know, it's, it's still something that you're like, can we, can we enforce this? Can we look at this as something that we use as a North Star? How do we work to chain control into this whenever there's new tools in use, things like that? But most environments, I see a lot of ungoverned Claude, GPT use. I'm sure you may see the same, but that's kind of like a big forefront where there's a lot of data leakage going on today.

Yeah, we were in a—

oh, go ahead.

Uh, we were in a, we were in a situation where we weren't exactly sure where, like, we should— which platform should we bless, I guess, you know. The big ones are, you know, ChatGPT, Anthropic, Gemini. We already have like Google Cloud stuff, so Gemini was like a first runner, you know, first test bed. And ChatGPT, people were starting to use it and we were seeing that in logs. And then we kind of made a decision to consolidate onto Claude for a few reasons, because it happened to be the best at the time. We knew the things to expect because it was a single platform. So after that inventory, perhaps some type of, you know, this is the blessed path here because people are going to do it either way. So perhaps giving them the easy path, golden path that is sanctioned, is kind of like— You know, the next step.

Yeah, yeah. One of the things I wanted to chat about, I do love that you started with the policy first. And I, I think most MSPs listening today, they understand policy, they understand why we have to have them. They've probably written some themselves, they've probably tried to get many of their clients to get, uh, traction with them. And while policies are not enforcement in and of themselves, like you said, Mickey, like, I think sometimes people think, well, if I just write the policy, then everything falls into place. It's like, no, no, no. If you think about the policy like the Declaration of Independence right here in the state, this is our way of saying we are now designating and declaring, like Michael Scott, I didn't say it, I declared it. This is what good looks like. This is how we're going to do it. It grants authority to the structure. It grants definition to the structure. And then that is what allows for the governance and the control to come into place. And that's where we started the policy first, is that Policy doesn't— it's not just magically making everything operate correctly, but it defines what correct should actually be, and it gets the client into the conversation a little bit more. Now, I realize some clients are like, I don't know any of this nerd stuff. I just want to use some AI. Shut up and go away. That's a different discussion. We've already talked about that on Above the Stack already, of how do we get the client to understand the purpose for this. But I like that you started with the policy because I think that is the most important part. And then One of the things you mentioned right after that, I wanted to ask this question to you, Mickey, is— Mikey, my brain is not working today. I need to like— someone needs to like, I don't know, smack me in the face every time I say Mickey. Mikey, one of the things that I wanted to ask you was when it comes to— you mentioned like you all said we're going to start with Claude because at the time it was the best. How did you handle that hurdle of people being like, wait, you're going to take away my favorite toy? Maybe I wasn't using Claude, but I was using Gemini or Grok or whatever, and now you're telling me I can't use it. How do you handle that situation?

Well, I can say that it was painful. I guess some practical AI advice is to understand that all of these platforms, especially the big 3 that I mentioned and a few others, they are like a rented environment. Think of if you're a marketer and you use social media, you don't own what you write on Twitter or LinkedIn, they do. The same thing is true for these big AI providers. The more that you can have a local context and memory and use your hard drive, everyone's got one on their computer, use it to create the context that you're needing for everything that you're doing. But yeah, that was a pretty big lesson pretty early on. So we were trying to create documentation around things and then use that as like a company-wide, as like a, you know, context boilerplate. And we'll have like, you know, weekly or monthly kind of training sessions, like this is how you do it.

Mm.

And that, and while that stuff was helpful, it did quickly fall to the wayside when, as models became better, as the harnesses became better. And like, this is a very, very moving target. So don't feel bad if you feel like you're behind or anything like that, because it is moving very rapidly. But you're going to learn things every day, and that's no reason to stop trying to be, you know, perfect or strive for, for the optimal thing that you'll have to tear down in a month, because that's likely going to happen. But just practical AI advice is keep as much data memory outside of the platforms as you can so that when you switch, it's not as big of a deal. And we had to do this recently again when we, we like split some Claude stuff, like team accounts. So like half of the team lost all of their their memory essentially that was buried inside of their cloud apps or whatever. And again, the painful lesson resurfaced that we didn't learn well enough and had to learn it a second time. So just a word of advice while you're out there educating your clients, one thing to educate them on is to use your hard drive as the memory bank.

Yeah, I think that's a good point. And I think it's, it's important to call out to like, It's better to start earlier, right, with ripping that Band-Aid, if you will, on, you know, just getting people into a governed lane. And, you know, you get a lot of both obviously security benefit, but you also get a lot of, you know, productivity benefits if people are truly using shared spaces or in Claude, you know, you can use a project or something like that where there's both company context that's being, you know, worked on by many users or a team or department, but also— In a broader sense, like, you know, you get a lot of, you know, benefits in people using it to create business outcomes, right? In that sense, versus living in a silo and pumping their data, the corporate data into, you know, a model that's trained by, you know, the larger aspect. And Anthropic is an example with that. Did you guys, whenever you rolled out, you know, that, how did that look in the sense of just you know, kind of a crawl, walk, run, you know, with giving people access to connectors and plugins and things like that. Just be curious to how you guys tried to kind of navigate that because it's hard because everybody obviously starts asking for more and more, you know, as they get access or see something online or, you know, see it even native in Claude make it really easy for you to connect to your Gmail, right, as an example.

Yeah, when you, when you start down the road of connecting your AI to the tooling that you have, let's Let's say like everything is basically a database, your CRM, your support system, your HR system, finance.

Mm-hmm.

Like all of that is a, a system of record that is essentially a database. And down the path of connecting those systems to your AI tooling is where you have to really be careful to use best practices as like a, you know, security concerns. And I think the most powerful thing you can do is get some type of MCP gateway of sorts. There's a few open-source versions that you can host yourself. There's some great products out there. The one we use is called Mint MCP. It's really good. When you're connecting all of this data that you really don't want to get out to the outside world, you have to be very stringent on who can access what. Everything has to be identity provider locked. And you were mentioning that, Nick, too, like that identity control is consistent all the way down, down the chain through this AI journey. And then you have to be careful with the tools that these MCPs can access. So like crawl, walk, run would be, you know, connect your CRM, but have it be read-only, only allowed to people who have an account with that CRM. So like, let's say that's marketing and sales. They have read-only access now to that CRM data in their AI tooling. And then you can broaden that read-only access to say finance executives. Then, you know, maybe engineers never need to see that, which is totally fine. And then you're— one day some team is going to come to you. This happens to me like every week that says we would now like write access. to the system, to Salesforce or HubSpot or one of those type of things. And then we have to figure out the capabilities that we allow, and we have to very carefully scope, you know, when you build like a client app or an API key, you very carefully scope what it is allowed to do. And we are not afraid to give the most strict scoping that we can. And assume that maybe one day they're going to come and ask for more scopes, more privileges, more permissions in that system. And the benefit of this is manifold. You get the users the data, like our team is, is just performing exponentially at the moment. But when you give them the right capabilities, like they're able to use like a one-screen interface, their AI tool, to do the work now. And you can also— don't tell the big vendors this— but you can like extract the user count because now not as many people need licensed accounts in that system. Some people just need to read it. Some people just need to do action on it but don't need to see the web user interface anymore. And that can all be created and maintained through some type of MCP gateway that gives you a big— a pretty big powerful tool to implement. That's like, I don't know, step 4, 3, what are we on? Governance.

Yeah, this is what we're talking about right here, all of these things. I think you're really answering these questions well. One of the things I want to just come back to the audience here is that part of what's important is the same things that have always been important. Remember how we used to harp all the time, Nick, on data flow diagrams and understanding where data lies, How data is accessed from one system to the other. Like, these are really important things. And I think what Mikey is correctly doing is helping us understand the connection between what should always have been good security practice into how does data flow inside of AI? How do we wire and connect systems up? How do we authorize the right users to get access to the right things and not the wrong things? And how do we do this in a way that doesn't inhibit innovation and growth and speed as a company, but also does it with structure across the way so that we don't just have anyone, you know, going right into, you know, a Claude agent and saying, hey, what's Nick's salary here? And tell me, you know, all the things, all the HR actions against him, et cetera, et cetera, et cetera. So I love that, Mickey. I think that really is an important point that we really need to make here.

Yeah, you mentioned a really good point there, Wes, where you're talking about innovation and experimentation. So like if you are too strict, that there is no room for that to flourish. And that is how you figure out the next things that are possible. Like I'm sure you guys every day are finding new uses for Claude or whatever. Like I had it edit a video for me the other day in DaVinci Resolve, and then I had it like extract a 30-year-old t-shirt design and make a vector in Illustrator. Like it's getting a little insane of what these things are capable of. So if you can't, you got to clamp down. And we hadn't gotten to really controls yet, but identity is one of them. But you have to allow room for experimentation.

Yeah, I think that's so huge in the sense of both, you know, letting people explore, because a lot of ideas I think even that come up too is like expectation versus reality type of situation where they might get their hands on something and figure out like that's not the problem they need to solve. So there's a lot of shiny object syndrome that can be alleviated through giving an approved like sandbox, if you will, for these types of things. But I do think the core security aspects are so powerful, you know, when you think about this and think about it may be very intimidating, but if you go back to your core basics of identity and access management, least privilege model, you know, the data flows, I think, is probably an area where we're a little bit less experienced because we didn't have to deal with this level of, you know, data being exposed to users and AI and obviously You know, just the implications of giving write access are really scary, you know, to these LLMs. When you talk about like pointing it at our CRM, I get really freaked out about doing that. When you talk about like, let's mass update 3,000 records, right? You know, using the CRM or using, you know, Claude or something like that. But, you know, all those are big components. I think the, you know, just having a foundation in place for our SOC, audit too. We did like a risk registrar, and that's been around for, you know, years obviously as well. And it's just something that we're saying, hey, we understand what connectors are in place, we understand what risks are exposed because we either don't have the full licensing with this product or, you know, because we just are accepting that risk, right? Because it's low to us because of the ways we operate. So there's a lot of things that you can do, and I think it's important also from observation just to not get into Overly restrictive where you can't even fit, you know, when you're writing that policy or you're thinking about like implementing things. Like, obviously you don't want to be so loose that it's just still Wild Wild West, but I think if you put too many restrictions, you're likely to not follow it and lead anyone. And so it's kind of like that crawl or run is another observation. I don't know if you guys have seen the same, Mikey, or, you know, do similar things in that sense.

I'll give you a story that will probably scare you, Nick. So every company has some, some type of HR system with everyone's Social Security numbers and salaries and like a lot of very, very sensitive information. Well, our HR team would love for that to come through Claude in our case. So we had a very frank discussion with our security team. And we identified exactly the things we were going to allow, like if you think Like my dependents, my son's Social Security number is in there. And I don't even want Anthropic to have that. You know, I don't know, they probably already do, whatever. But it's— there's this give and take, and you have to err on the side of take away first. So like this particular system, HR system, it can do like 3 or 4 things, and they're read-only. And then what we did is include a middleware that prevents encode and does even does like it has like a kind of backstop regex that will look for things that look a certain way. Social Security numbers, salaries, birth dates, DOBs, that type of stuff. And the middleware gives the people in HR a very specific window through an identity group. So that they can see certain things. Everyone else gets a different window that is even more strict. And then there's some hard regex limits that, you know, I understand that's probably brittle and will change over time, but it gave us the confidence to release such sensitive information into our AI systems that we were comfortable doing it and even got security sign-off. Which, if you've met our security guy, you would be like, wow. Hey, Mike.

I do think though, this brings up such a great conversation of like, how do we avoid shadow AI? We've talked about that already on the show before, like what it is. If you think about like shadow IT, there's 2 reasons why. And just so everyone knows what we're talking about, we say the word shadow IT or shadow AI, we're basically talking about unauthorized use of certain tools in order for someone to get the job done. In my experience, I don't know about y'all, but there's 2 reasons where this typically comes about. One is the business has never enabled the employee in a way that they feel like is easiest, and employees are like water, they find the path of least resistance. So they're not trying to be evil, they're just trying to get a job done. That's the first reason why it comes. And the second reason why it comes, and I have seen this from overbearing security people, that Fail to offer a better incentive. And so sometimes, as— and I'm a security guy, right? Like, that's my background. But sometimes if all we do is we want to revoke and pull back without finding a way to offer an alternative, even if it's not as good, they're going to circumvent the system. That's the second reason why it would happen. And so I just want to pull on that thread a little bit more, maybe to both of y'all, but Mikey, I'll start with you. How do you avoid that? How do you encourage proper use of AI tools? Obviously, we do these things like you mentioned. We trim away. We use, for example, DNS controls as a good example of some of that and some other tools that we can use at the identity layer, at the machine layer. But like beyond just the control aspect, how do we start to introduce the right practice that avoids the desire for the employee to want to go put shadow AI in place?

I've noticed some chat from GetSecured from looks like YouTube. And he started off with a good one, that AI proper use goes hand in hand with education. And that is what I would say to this. And what we found is, as we were having like our monthly how to do cool stuff with AI talks, is that a lot of people were— they were kind of scared. Like our employees, our staff were a little bit nervous giving— I guess they felt it was like some power, you know, power-up, which it is. And they were just a little bit nervous that they were going to break something. We've been starting over with fundamentals training on these AI capabilities, which turns out are not that much different than what we were teaching before. You mentioned people will find the easiest path, that is 100% true, and AI does the same thing if you haven't noticed. ChatGPT hacks Hugging Face because it was like, oh, I've got a goal over here, Oh, this— oh, I can, you know, sidestep this system and get out to the internet and do all the things I need to do. Like humans, we're the same way. We're going to find the easiest path. So you should have like a happy path. I want everyone to do this this way. But you should also have some ancillary flexibility, as long as it's not too outrageous.

Yeah.

If that's an answer.

Yeah, I think that— I totally agree with that. I think, you know, education is definitely a big piece. I think that what I mentioned earlier around, like, peer enablement is also good, because if others see their peers using it and they collaborate together, they're much more likely to adopt it as a whole versus individual use cases in that sense. And then sharing wins, right? It's like you've kind of built a culture around this internally where we are, you know, able to use it and use it securely. And we always say any tool you use, you're going to have frustrations, whether it's Copilot or Claude or, you know, GPT, anything like that. But there's also considerations, you know, for individual users to get, like you mentioned, I think not many organizations are doing like internal AI council or, you know, just kind of AI champions that are more of your power users that can share learnings, can do experimentation, can help, you know, the organization do the training, you know, versus just solely being on one person's hands or anything like that too. So I think there's a couple of different, you know, integrations there, but I also think that stems from the top down in the sense of leadership. You know, it sounds like for you guys, like, it was an initiative to build this whole department that you run now. And, you know, I think that's a big piece to it too. You have to have that bought in or else people are going to go, well, what is the direction here? You know, what do we use and how can we, you know, maximize our role? Because all they're trying to do, Wes, to your point, is Not trying to be malicious, they're just trying to get their job done faster. That's why they use shadow AI tools or shadow IT tools in the past. And this is even more accessible, right? It's more marketed to them and it's right in their face with every tool they use. Copilot, especially if they're using Microsoft, they're spending a ton of money to shove that down your throat. And, you know, it's going to be in the forefront of every app that you use. So it's important to recognize.

I'll give you another story that's kind of related, and perhaps a solution to some of this shadow AI, is that we had a sales guy, very, you know, entrepreneurial salesperson, trying to get up to speed on like, uh, responding to customers, his, you know, uh, people that he had to reach out to and coach himself on, um, how his calls were going. So, uh, he had a friend build a— they just did this You know, and gave him access to load a transcript of a call and give him a report on it. And then he comes to me and says, is this allowed? And the answer is absolutely not. This is, you know, this is not something— this is data exfiltration. That's the name of this. So what can we do about it? Instead of just cutting him off at the knees, we, we actually built a simple plugin in Claude that can do the same thing for him. And now most of the sales team uses that to coach themselves on their call cadence and understanding what the client said based on those transcripts and then helping them with follow-up, giving them some pointers and things like that. So don't just say no, provide the other way, the solution that you are blessing.

This is such a good conversation thread.

Go ahead, Nick. No, I was just gonna mention, we've definitely, I think Wes on the last episode or maybe it was with Dustin, we heard similar conversations, right? Of like, hey, you know, we investigated this thing that you're trying to get done and maybe it's just a better process, right? Not even using AI that we could help you solve for that you didn't even know existed with basic technology, not going extreme to LLMs or something like that. But I think no matter what, they're The switching costs are so low between the tools, like there's not going to be an exponentially better experience necessarily, right? Depends on some use cases, you know, obviously I can't say that in a blanket statement, but I can say universally for the basic workflows that most people are going to be trying to use it for, or automations, like there's not going to be much variance between the frontier models and the tools that you're using in that sense. So it's a, it's a good call out for sure.

It is, and it brings up this who question, I think, that I want to shift our conversation into just a little bit. Typically in the world of like classic enterprise IT, you know, we band our users based upon role access groups, based upon job function, what they need to know, what they need to have to accomplish the task, all these kinds of things, right? So we might build in Entra or Active Directory, here's an organization unit that can do these things, and in firewalls, here's access groups that can do these things. We begin to think through it from that perspective. And I think that largely kind of plays into how we use AI. But I think there's— it gets a little bit more nuanced. And I'll give a couple examples of how I think this gets nuanced, where, um, Mikey, you were kind of getting me thinking on this. Is one area it gets nuanced is you do have classifications of some users. Like, you're always going to have some users that are like, look, I'm not opposed to using AI, just tell me exactly what you want me to Like, you're just going to have those regular employees that are not thinking outside the box. And for them, you know the quote, any technology sufficiently advanced is indistinguishable from magic, right? And so they're like, they don't even know what an MCP is. But if you give them this place that they can go hit and they can ask a question and just say, tell me all of our high churn risk customers that are in this area that we haven't had, you know, this criteria, we haven't talked to them, whatever, blah, blah, blah, and they get their data back. Wonderful, right? Like, that's good for them. They're like, they don't even know it's AI. They just know they can talk to it in a normal language and get answers back from data systems. Wonderful. But I also do think you're going to run into some of those people— I usually call them the kids— that are like so wickedly smart, that are vibe coding, you know, at home on their own, that are doing some really, really, really cool things, that think outside the box, that don't accept status quo. And as much as they have the capability of doing freaking awesome things, They also have the capability because they don't know what they're doing of literally taking a network down or literally exposing critical enterprise data. And so that who question I want to bring up to you, Mikey, how do we incentivize those kinds of users without also having them destroy everything that we've built in one fell swoop? Like, how do we manage that balance?

Yeah, I think the policy is like the document that binds everything, but the controls are really the thing that does the work. Unfortunately, we need to protect ourselves from ourselves, and that's what we do. We can give you access to our CRM, but based on your permission level already in that application is exactly what you're going to get, probably clamped down to read-only at the beginning to get you used to it and then maybe escalate you to more write functionality in the future. But the controls are what keep us contained. But our controls are in such a way that you can really do some impressive stuff. Like, we've had people come work here and they're like, oh wow, you guys like let us do everything with AI. And we feel like we're even strict. So, which I guess that's a good balance to have. Like, we feel like we're clamping down really hard, but they—

Yeah.

employee everyday work feels, um, broad, like they have broad, uh, scopes and everything they, they need to do. So you have to, especially those like very experimental A-players or whatever you want to call them, like you need to give them the room to run. So like we do things like you can only use an MCP from our blessed gateway, which may sound like, oh wow, well that's terrible, what if I want to use a random thing from GitHub? It's like actually that's exactly the thing we're trying to prevent. But tell me, what do you want to do with that thing on GitHub? Maybe we can recreate it and put it into our gateway, which is super easy to do because AI writes the code. We just have to give it like a specification and permission levels and then combine that with the credentials necessary from our IT department. And now that capability is available to you and we didn't have to download any random code from GitHub. No offense to GitHub, but you know what I mean? And the turnaround on that type of stuff is like 24 hours. So like if someone comes to us and says, I want to use this, but we can't— I can't because we've restricted access to our MCP gateway. We can turn around and give them that capability in a way that we consider safe in about a day. And if you can, you have to be able to operate on that speed. In order for them not to feel constrained, I guess, is a good way to say it.

That's, um, I think it's—

that's interesting too, Mikey, too, because I was, I was curious, like, the way you've been talking about this, but also just kind of that motion, it's like, I'd be curious to understand, like, how do you, how do you manage the throughput, right? Because that's kind of the next logical progression of, like, probably getting requests a lot from users. And you guys, I think you mentioned, are like 150 employees, um, or so. So You know, like as you scale up, obviously there's more and more noise, there's more and more requests. How does that look for you guys in the sense of like a change control board or, you know, what kind of controls do you put around, you know, just that process as a whole?

So right now it is more manual than I would like, but I did just hire someone to join me in AI Labs who was just published in Wired, which is pretty awesome. He made a thing that does like Grand Theft Auto 5 and adds a flock camera mod to it. Wired picked it up today. I was like, that's awesome. Anyway, so he and I are going to, you know, operate like a support team. That's kind of what we're doing now. Like, you know, message comes in, Mikey opens Claude, does thing, gets credentials, you know, deploys, tests, and then makes it available. That's about 24 hours. But I think we can do even faster. We've been working on some software factory automation where the message doesn't come to us, it comes to our bot that lives with us in our chat application, and it takes the request, adds it to a board, and then if it can build it, it just continues to build it. We've been in the early stages of testing this already. That's how we can handle the throughput. But honestly, just like you in front of the keyboard with all of the context, that is available, um, in, in the way that we work now, it is still very, very fast. So like a day turnaround is pretty average, um, but in the future it might be like, you know, 30 minutes. So that's our, that's our goal in the near future.

Got it. That's pretty impressive. That's, that's quick.

That brings up another question. Um, maybe it's a little off topic to this, but I think it's related enough to where I think it's a good It's something I'm curious to know. Let's talk about, like, if I'm an MSP today and I'm starting to get a few clients that are really adopting AI and they're really getting comfortable with its power and they're really getting excited about it, one of the next things that starts to come up is token spend, right? So they start to think, well, what employees get authorized to have a budget to be able to use and burn tokens down and drain data centers and lakes dry to be able to do something kind of awesome? But that doesn't mean they're efficient with it, right? One of the things I think we have seen is immature usage of AI that leads to literally burning down tokens. I think there's a conversation that needs to happen there too, that just because I can go to Claude and I can ask it to go pull tons and tons and tons of data down from the CRM and do all this complex stuff to get me this report for management, I just burned $100 in tokens in a simple ask, and there probably are better ways to do that.

Yeah.

do this, where we leverage AI to go query data that's maybe in a SQL database that's built somewhere else or an MCP layer. That's— so I just, I think there's a conversation there that needs to happen too, because if we don't start doing some of these things, we're going to see the increase in use of AI doing awesome stuff, but we also may not see a reduction in cost. We may see an increase in cost because we just have people who don't know what they're doing. Is that something y'all are also seeing over there at DNSFilter?

Yeah, very, very top of mind here at DNSFilter is token spend. Um, first of all, the, the big, the big AI labs and their subscriptions are highly subsidized and are your friend, but at some point that will not be the case. That cannot last forever in our opinion, my personal opinion too, but you have to prepare for a world where, you know, those Fable tokens are $10 per input or whatever, $10 per million. So there's a few things you can do. So if you think of the systems that we have connected to, like our CRM, HR stuff, Binance stuff, alerting systems, Grafana, things like that, you know, those APIs are well documented. And it is not that much work to give the AI the context it needs to query that system efficiently. So like we have a huge data lake with every piece of data we've ever created. It seems like I can't find anything that's not there, but it's very expensive to run. So we had to spend a lot of time and our platform team actually did this. created a set of skills that give that system super efficient access to what it's trying to get. It's like you can't just jam 10,000 lines of JSON into a model and expect it to perform well. It's feeding back with the understanding that AI is getting this and it needs to be able to use it efficiently. Creating let's say fine-tuned skills that make your connections efficient is one way. The other way is model choice.

Mm-hmm.

So there's a lot of talk about model routing, like making it automatic would be preferential, but not quite possible yet easily. But you need to educate your team on like, are you checking your email? Like maybe that's the lower tier model is just fine for that. And we have a cadence where we educate people on best use cases for certain models. In our case, Claude. So we're looking at Haiku, Sonnet, Opus, and Babel, and, you know, divvying up tasks, you know, examples in fact of this is the type of task that Haiku can handle. And we're trying to get people prepared for a world where burning tokens is not as cheap because it's coming.

It is, it is coming.

Nick? Yeah, I think it's super hard. It's a very hard problem to solve. I'm glad you brought it up too, Wes, because it's part of the security equation, I think, in a lot of ways too. Because if a user gets compromised and you haven't set budget limits or things like that, then you could get the same thing we used to have with, or still do have with, people spinning up Azure farms of crypto mining and things of that nature. But I think it's just a hard problem given, like, we're trying to make user a power user, in my opinion. Right, by giving them the controls to say which model do you want to use, what effort do you want to use, and all these things that they're— nobody's going to ever click on and use as an average user.

Right.

And they're not going to know what Fable from Opus is, right, as an example, and when to use that. So coming back to your point on education, I think is good in that nature, but it's really cool to hear you guys have done these other types of controls or wrappers, if you will, You know, just to get people into compliant workflows where you know that you're maximizing the usage for the outcome that you're trying to drive. And I think that's a cool approach to do that as well. Just given you're kind of trying to premeditate, you know, everything— not everything, anything needs to have Fable. In most cases, it doesn't.

We have like a— I do like a dashboard every week that's like the cloud usage You know, going forward or in the past, and it's like I'm always at the top of the list feeling terrible about burning tokens. But I also have like multiple accounts and you know whatever. But you have to be conscious of of your spend and and start educating yourself on model correct models to use. Be your own model router.

Yep, we're in that same boat. You know, as a software company, all three of us. Like, we, we understand this, right? And we do all kinds of things. We even look at token arbitrage, and we look at like— we're— we've, um, we do play between different models to see which ones tend to be more performant. We even play around different frontier models and we experiment with those. I don't know about y'all, but right now we've had a lot of luck with Groq. Groq has been actually by far more efficient and more capable than Claude. We're now at like 98% of all PRs are fully written by AI with human in front of it, so we're not just vibe coding. Like, we have humans that are fully driving it with a QA process behind it, but it really has changed the game for us. But now all of a sudden what's come to the forefront is token usage and efficiency and capability and making sure that we're doing this the right way. It really has changed sort of the strategy of what's happening. It's almost even become too where, you know, your regular employees almost become managers themselves, right? Because they have a whole host of agents.

Mm-hmm.

And so now they have to almost think about, how do I manage my agent as if I manage my people? And that goes back to that discussion that GetSecured had mentioned in chat around education, right? Is now that we have to think about how are we building our agents and are we efficiently building them? Even doing things like some people call agent looping and like an agent that manages other agents and looking over what they're doing and finding more efficient ways and more secure ways to do things. Like, it is wild to think about this is the world that we are in today. But, you know, with more power comes more responsibility. And I think that's one of the big themes of this discussion today, I think.

Yeah, I think it's important to give— since you mentioned agents, give an agent an identity too. And there's, you know, there's kind of like a spectrum of is GrokBot an agent? Is GPT Codex an agent? And like, in a sense, yes, it is. In another sense, there's like way more automation that can happen on the other side of that spectrum. But either way, the agent itself, especially if it's autonomous, needs to have an identity with certain permissions that— and a budget that can't be circumvented.

Yep.

So as we kind of finish up, one of the things I want to show this slide here is what should an MSP be doing next when it comes to working with clients? Like, what kinds of things should they be thinking about moving forward on? And I love this first one that we had here is, what's something we can do within 30 days, right? Like, what's something that we can do and what relatively immediate that they can see the ultimate results, the ultimate impact? We're not saying this is going to change the game, it's the most complex thing, the most expensive thing the business has done, but like, what's one thing that hurts? And what's one thing that hurts Smarts that we can solve relatively quickly. I really love that. Is that kind of how you would start this journey, Mickey? Is— is— God, right when I thought I was done with Mickey. Lord help me, my brain is not working today. That's what I get for having travel brain. I'm in, uh, I have a client meeting after this. I've been early morning. Mikey, Mikey, Mikey, Mikey, Mikey, Mikey, Mikey, tell me, is this how you would start these conversations if you were in the MSP seat?

Yeah, I think just running that inventory kind of step, just showing the customer exactly what's happening in their environment. And you have to have access to some DNS logs. And if you just have, like, you know, raw logs, you could probably feed that into your AI tooling and have it pluck out, this is an AI domain or something. You could probably do that, but just show them. The amount of AI tooling that is being used in their business that they might not even be aware of is probably a good way to get the conversation started.

Yeah, we see that a ton, you know, at Cloud Capsule and just like talking to a bunch of MSPs is either you have a business stakeholder that says, yeah, I definitely know that, you know, there's some shadow AI used, or they're vehemently defending They don't, right? And so we replaced it with saying, no, no, we invested all this money in Copilot. We're using it every day. And, you know, it's— then you show them both adoption of Copilot in the shadow AI, and it's like, well, actually, most of the users are leveraging and uploading and downloading a lot of data from it, as well, too. So I do think that's— it's definitely a gateway to just kind of getting the conversation going. Of just, hey, you know, this is why we want to start putting these things into place. And again, to your point earlier, Mikey, just around, you know, not blocking usage, right, but maybe putting into a governed lane, you know, in the sense of the approved toolset, right, a sanctioned tool. To your point earlier, I think it's just the appropriate path. And then trying to bolster the education around— I think that's the opportunity for MSPs today, obviously, is just what you've been doing well in the past, which is security and governance. as a service, but then complementing it with workshops and education around how to use this in a productive sense and getting the most out of these tools. There's certainly a lot of talk checks there which will drive compliance. You know, it's kind of a byproduct of why you should invest time into it. That's part of it too. So the education piece is, I think, a huge opportunity, you know, because it's just like you need an advisor. Your business doesn't revolve around these AI tools. You still need an advisor to help you guide you through that.

It's like the great reset of being the, uh, the advisor in the room. Like, MSPs get a second chance to be that advisor. Like, here it is, let's take the opportunity.

Yes, you nailed it.

Yep.

And as I think that's one of the themes here is being a great advisor, being a great educator, helping them understand the things that they don't know. What a wonderful place to be in. For sure.

So just want to tease this again for those that came in late.

One of the things that we are giving away today is a securing Claude guide. This comes over from Nick in Cloud Capsule. And so Nick, just want to say thank you for that. The QR is right here and we'll move past the slide in a minute, but we'll keep it up over there wherever it is on your screen. On that QR, you can grab that. A totally free guide towards how do we start thinking correctly about securing Claude as we start to move towards it. And even if you're not using Quad, go give it a read. It's still really valuable to understand because a lot of this can be crosswalked to other models as well. Mikey, tell me a little bit more. Anything you want to tell us about what's going on at DNSFilter?

Well, this AI world has been pushing us very, very fast. Like, we— like, our CEO is back into the IDE again, you know, building things that actually you know, not just like junk code like you would think a CEO makes, actually useful things that are pushing our product forward. We are looking at how to identify AI usage even more than we already can. So things like local AI tooling installed like Ollama or BLLM, like that stuff running. And we are definitely running like dogfood betas on our own systems. to see, like, what we can pull out of the device to understand what is happening there more. So we're looking to illuminate the world of AI that is in your company even further than we are now. So stay tuned.

That's super cool. Yes, if you want to give Mikey a follow, there he is, Road to CISO, which is a super dope name, by the way, in LinkedIn. And go give the awesome, incredible folks at DNSFilter a follow and check out what they're doing. They're a great, great company. Coming up next, it's really funny how we inadvertently sort of stumbled into why is it that education and learning and enablement is such a big deal. It just so happens that that is the next episode. So October 22nd, 2 PM Eastern, mark those calendars. We're gonna be bringing on Andrew Moore from Ridgeview. This is gonna be a really fun discussion. Hopefully, Mikey, I won't just keep butchering his name like I've done to yours today and embarrassing myself. But this is gonna be a super fun discussion. If you don't know Andrew, he really specializes in this world of how do we help build towards not just education, but how do we transform through enablement and knowledge? And so how do we get an AI learning strategy in place? It's gonna be really, really, really fantastic. And that's all we've got. I wanna say, Mikey, thanks for joining us today. It has been super fun having you on, super insightful.

Thank you.

Really appreciate you joining us today. And everyone that's been listening, give this a follow, give it a like, all the normal stuff that we always say. And until next time, thank you guys so much for joining. Hope you all have a great rest of the day.

Thanks, guys. Bye.