008 AI Security Do's & Don'ts

Oh my gosh gang, this AI stuff is so cool. But if it's this black box, how do I know that they're not running off with my sensitive lucky charms and sharing my client data? And what stuff can I put in here versus not put in here? Let's talk about it. It's kind of a spooky new paradigm and how we manage our data. But it also has a lot in common with I think the best practices that we should just be using in general. So let's talk about do's and don'ts of AI security. And we're also going to talk about what I think is like the biggest distractor for us in our day to day that pulls us out of like that kind of deep focus. So let's talk about a spooky AI stuff today. So what got me here was talking online about that, that bank transaction extractor deal that I built, I don't know, last Monday or something. Basically the idea was GPT as it turns out is actually really good at extracting bank transactions from bank statements. So if you have a client bank statement and you don't have a transaction CSV, you can copy the text out of that bank statement, paste it into GPT and ask it for a CSV format of download of those transactions. Does it really well, even like the free version of ChatGPT which runs the old GPT model. And my experience did it flawlessly. And the reaction from a lot of folks were how should we be thinking about the sensitivity of client information if we're chucking that stuff into GPT? And it's absolutely something we should be thinking about and talking about. Anytime you get into data security, people have different, different thoughts on which things are confidential information and which things are not confidential information. So this conversation is a little bit of a moving target. And there is an aspect of it that's squishy where you're just gonna have to make your own determination and say here's where we draw the line. But there are foundational aspects of it that I think are easy to understand and you can kind of make your own decision on top of that foundational information. A good example of this squishiness is a transaction line sensitive information. Transaction lines contain, you know, the merchant information which is the same for everybody. And then a dollar amount. If you have that information without any information about the client, it's anonymized. It's just that block of information. Is that sensitive info? Some people say it is, some people say it isn't. I don't know that I have an absolute answer. I don't think there is one. It's just something you'll have to come to your own conclusion on and, you know, kind of build your own policy. But what are some of the more sort of foundational aspects of AI security that are kind of just good to know? Because I think what I'm seeing now is similar to when people went through the desktop to cloud transition, where there's an element of spookiness and maybe what's hard to understand about it makes us a little more hesitant to leverage it just because it's all still kind of ambiguous. So a couple of things that I think a lot of people aren't aware of and I don't want to sound like. I try really hard not to sound like pro AI guy, because fundamentally I'm not pro AI. The whole like open letter the other day with putting a six month pause on AI, like there's a lot of merit to that. From a practical standpoint, getting everybody around the world to hit pause on that thing for six months seems kind of impossible. But AI moving too fast is an issue, and not through the lens of some sci fi, you know, enslaving us sort of thing, but the idea that if a model comes out tomorrow that genuinely displaces a lot of people, like the economic whiplash of that is scary. Like we haven't navigated something that's happened that quickly before. So there are aspects of this that we have to be mindful of. The arms race of creating a more powerful model at any cost, like the energy requirements of that, like what that means for us socially and ecologically. Like, there's a lot of things tied to that. So I try to steer clear of like whole hog sounding like pro AI guy. But AI is such a divisive thing right now and people are very opinionated. And honestly, I think that's the easy thing to do is to form an opinion and take a side when in the reality is like nobody knows exactly where it's going to take us. All I can do right now is try to keep educating myself on what's happening and try to leverage it responsibly. And so like, I try to limit my perspective to just that. What are the responsible ways that I can do it right now? And I'm gonna keep educating myself to make sure that I'm kind of, you know, on the leading edge of being able to use it in a responsible way. So getting back to some kind of foundational truths about AI security and what's out there right now? One of the biggest, like boogeyman type things we've seen in the past is the notion that your prompts for, for a language model like ChatGPT are what train the model. And in the case of OpenAI's model, the GPT models, this used to be the case. So when a user types in information or pastes a bank transaction in or something like that, that information and the feedback a user would give on the response would be trained back into the model to, you know, improve the model. That was part of how the model is developed. And so understandably, the notion of it knowing the stuff that you tell it, even though there's like an incomprehensibly large amount of information in that model, was a little spooky because theoretically Steve could go out and prompt something about it and some fragment of that thing that I told it could resurface, right? Like the likelihood of that actually happening, like probably infinitesimal. But the reality is like it was being trained on that information. Now, as of March 1, OpenAI has stopped doing that. And so in fact I can pull it up here, their privacy policy. So starting on March 1, we're making two changes to our data usage and retention policies. One, they will not use data submitted by customers to improve the models, and two, any data sent through the API will be retained for abuse and misuse monitoring purpose for a max of 30 days and after which it will be deleted unless otherwise required by law. And this looks more or less like the retention policy you'd see with most of the apps that you use. So specifically with language models like GPT and stuff, you're chatting with the, like the best practices here. You're just going to look a lot like the best practices for any cloud apps that you use. So you got to understand the data retention policy. Are they selling that data specifically with language models? Is that data then being used to train the model? Currently with OpenAI models, that is not the case. But there's a lot of language models out there. If you're using Google Bard, is that the case with that one? If you're using Bing, if you're Google searching, like, is that information being trained into the model? If you're going to use a tool, you got to understand where that data goes. And in some ways that's common sense, but it's kind of been mystified a little bit by AI and kind of the black box nature of it. The reality is you gotta look at the reputability of the company Do I trust this company to go by what it's saying? Obviously there's a huge spectrum of legitimacy of software companies out there, so you gotta pick who you're gonna trust, but you gotta look at do I trust what this company's telling me? And, and do I understand data retention policies and how they use my data? And is all of that okay with me? So if you're working with an AI vendor, you gotta do the same due diligence with them that you do with any of the other apps that you use. But it's also worth like thinking through, like what are the standard best practices anyways around any of the apps that you use? So stuff like anonymizing data wherever possible. If you're going to park something in a database, do you really need all the identifying information of that client? Or could you use a client ID instead? Or, you know, in the bank statement example, if you're extracting transactions from a bank statement, you could select all of the text on the page of the bank statement, or you could just select the transaction table from the bank statement and paste that into GPT. Obviously one is fundamentally better than the other, so. So be mindful of best practices from anonymizing information and from never giving a system more information than it actually needs. One area, this comes up a lot with tax preparers, is when you've got a bunch of Social Security numbers and eins. In my book, I never stored that information anywhere outside the tax software. I had to have it in the tax software because it was required for doing filings. But since I like just the fact I had that data and maybe wanted to manage, you know, have a CRM somewhere else or something like that, that was not a reason to have to then maintain that sensitive information in other places. In fact, what I would generally do is just use a client ID tag for that other kind of peripheral information about the client. So if that information went into the wrong hands, maybe that software provider had a breach or something like that. The usability and value of that data is fundamentally different than if it was a complete record with Social Security numbers and stuff like that. But this is all kind of like common sense best practices. I think another, another thing that we do sometimes is we will say, you know, we're waiting for trusted app X to implement a thing. And I do think it's a bit of a misnomer sometimes if you compare, you know, like a company like OpenAI, who is far from something that some dude's running out of his garage to go, then and look at one of the software partners we use, that's really a small company in the grand scheme of things. Many of the companies in our accounting space, there's like an assumption that if they're in the accounting space they're fundamentally more secure. But particularly when it comes to implementing stuff like using language models aspects of GPT, at the end of the day those systems, unless they're developing their own language models, which is astronomically expensive to do right now, generally they're making the same API calls that are made when you use ChatGPT or something like that. So I, I do think sometimes we can kind of, it's kind of a mental trap to say, oh, I'm going to wait for this company that I trust to implement this thing when really if you run it through that company, it's probably just putting that information on one more company servers rather than going direct to open. I get it, you already have a relationship with them. It's not a new company that you have to like trust. But it's not as if that company has like a more secure back door than you do. I get that it feels different because it's AI, but for me you kind of got to like follow the same common sense framework that you use. Anytime you adopt a new app, don't give it any more information than it did. And 99% of the time it doesn't need the sensitive information to be able to do the job you want it to do. Hopefully that doesn't come off as like pro AI shilling guy. I do think there's some a level of spookiness around models right now that is unnecessary because with the reputable companies who are doing this, the data retention policies are very clearly outlined. They're not fly by night outfits. I don't trust OpenAI any less than I do many of the companies in our accounting space that are like 10 employees and much more rudimentary from like a software development standpoint. So that is where I'm at on that stuff today. There's some fundamental stuff like how data is managed in a language model that's pretty concrete. But then there's a layer on top of that where you got to decide for yourself what's appropriate for our firm and what's not. Have a policy for it so that your staff can understand what's inbounds and what's out of bounds even if you don't do any client data management with AI stuff right now, even if you decide that's out of bounds. But man, there's so much like internal workflow stuff that I can still enable. So the policy on both sides of that equation look a little different. But sometimes the easy thing to do is to, like, fall back and be like, I'm scared. I don't want to touch it for now. When with a little bit of, like, learning and digging into the privacy policies and stuff like that, on the other side of that, I think it's a little less spooky. Okay. And then the other thing I wanted to talk about. Okay, what is the. This is the. One of the most important questions in accountingdom is, what is the ultimate number of monitors? It's kind of like a hitchhiker's guide to the galaxy when they. They plug into the question of what is the meaning of the life universe and everything, and the answer finally comes back. I think it was after, like, years or something. And the answer is 14. That may be the, like, the most core universally relevant question to all accountants is, what is the proper number of monitors? And you know how I said it's easy to have an opinion? Buckle up, because I'm opinionated on this one, bub. So here's my career arc on monitors is you started with just, like, the single massive crt, right? Oh, my gosh. I just sound like the old person now. Dang it. Okay, so I started with one, went to four. In fact, like, everybody in our firm had four displays for a while, and we had to have, like, these custom video cards that were super expensive. And. But let me tell you, when clients saw that, they were impressed. And now I have, I think, ascended to this higher plane of existence where I'm now trying to minimize all that stuff, because feel free to disagree with me if you think I'm wrong, But if you've got more than two monitors right now, look to your right, look to your left. I bet you a dollar you've got your email up on one of them. You've got Team Chat up, you've got Twitter up, you've got my little face up on them right now. Here's the problem is the more monitors you have, the more you just park stuff up there, that will only serve to distract you. Right? Right. I ran the numbers at one point and was, like, thinking through, like, okay, if you've got a practice management system, you've got your email, you've got your phone, you've got Team Chat, you've got, like, I think I came to, like, 12 different communication channels. Your cell phone, if you've got all these different things and, like, you get, you know, X number of notifications a day on average. Like one of those channels is going to interrupt you like every two minutes or something like that. And the whole like point of me sharing that was you gotta close that crap down when you're not using it. There's a time for email and there's a time for getting work done because work doesn't get done when emails open all the time. All it does is distract you and pull you out of deep work. And hear me out gang. I think that's largely what monitors do for us. Now the exception here, the exception here is when you're entering data from one thing to another thing that absolutely happens, especially in like tax and stuff like that because there's no way to integrate those things. On the one hand, if you're entering a whole ton of data from one thing to another thing, that hurts. Like it's an indicator of like a lack of automation, lack of integrations. Could somebody lower skill be doing that for you? If all you're doing is plugging a number from there to there. But I get it, that's the world we're living in today. Sometimes you need two displays to be able to do this. I see you ultra wide guy flexing in the comments. I get it. You can split the screen. It's just like having multiple monitors. But I've come around to the idea that I think less is more here. One of my all time favorite things and I don't like to call anybody out, in fact I don't even remember who it was somebody, there's like an annual survey or award thing that's like here's the 10 best tech forward firms to work for in the United States per year or something like that. And there's like this census of all these different information and the software that they use and all these things. And one of the things on the census every year and they still do this is average number of monitors per employee. Like as if that is some yardstick of a future looking firm. Listen, I'm so monitor woke right now. I have bounced back from mega super whole ton of monitors to like now. I actually think like the less screen real estate you have, the cooler you are. I know, hot take because in accounting world it's usually been like who can put the most monitors on their desk. I'm like, that's where I'm at today. So is there an absolute correct number of monitors to have? I don't think so. It also depends on your role. Like if you're like super cool CEO shouldn't be doing the work, you know? Should he have any more than one display? Probably not, but I see you four display guy. You know who you are. Tell the truth. What is up on monitors three and four? It's nothing good. Unless it's this. It might be this, you know, YouTube channel right now. That's cool. But the stuff that's up there, I think, is usually just up there to distract you. So what do you think? Fight me. You disagree? Listen, I don't want to be opinionated, AI Guy, but when it comes to monitors, I think I'm anti monitor guy now. So on that hot take, thanks for coming and hanging. I'll see you tomorrow.