Unspoken Security Ep 40: Can My AI Be Hacked?

You know, the thing to take away is still software is easy to break. If you are building or using AI, take a step back and say, do I understand the risks of using it? Do I have ways to evidence those risks? If I don't, I should probably call some up and try and get a handle on this.

Welcome to Unspoken Security, the raw and gritty podcast for security professionals who are looking to understand the most important issues related to making the world a safer place. Including intelligence-driven security, risks and threats in the digital and physical world, and discussions related to corporate culture, leadership, and how world events impact all of us on and off our keyboards. In each episode, host AJ Nash engages with a range of industry experts to dissect current trends, share practical insights, and address the blunt truths surrounding all aspects of the security industry.

Hello and welcome to another episode of Unspoken Security. I'm your host, AJ Nash. Spent 19 years in the intelligence community, mostly at NSA. Been building and maturing intelligence programs in the private sector for about 9 years now. And I'm passionate about intelligence, security, public speaking, mentoring, and teaching. I also have a master's degree in organizational leadership from Gonzaga University. Go Zags! And I continue to be deeply committed to servant leadership. This podcast brings all of these elements together with some incredible guests to have authentic, unfiltered conversations on a wide range of challenging topics. It's not your typical podcast. It's not all polished necessarily. My dog makes occasional appearances. People argue and debate. We even swear here. I've certainly been known to. That's all okay. Think of this as the podcast. It's more like a conversation that you'd overhear at a bar after a long day at one of the larger cybersecurity conferences. These are the conversations we usually have when nobody's listening. Now today I'm joined by Dr. Peter Garrigan, CEO and CTO of MindGuard. He's also the professor in computer science at Lancaster University and fellow of the UK Engineering Physical Sciences and Research Council. He's an internationally recognized expert in AI security who's been dedicated for years to the scientific and engineering expertise. He's been putting that towards creating bleeding edge technology to understand and overcome growing threats against AI. So obviously AI, huge topic. Everybody wants to talk about it. And you know, so do I as it turns out. So Peter, anything you want to add to that resume?

Nope. I think you, you summarize it quite well.

All right. Peter's a man of few words, which is gonna make it tough radio broadcast. No, we'll be fine when we get into it though. I, I'll tell you the, today's topic, as you might imagine with Peter's background, is gonna be about AI. So the discussion today actually is entitled, can my AI be hacked? And so this is, this is interesting. Like I said, Everybody's talking about AI right now. AI is in everything, right? It's in our phones, it's in our computers, it's in our workspaces. Everybody's trying to do something with AI. They're gonna get rid of people or they're gonna make 'em more efficient or a little bit of both perhaps. But I don't think a lot of people are talking about the vulnerabilities and the weaknesses associated with this. I think I've seen a lot of people that are very quick to jump in and they play with some of these platforms and they're amazed at the output and they're just all in on it. They're sold on it. Oh my goodness, I, I needed a thing, I asked it to write it and it wrote this thing and it's amazing. And they don't really take a lot of time to consider how much we are offloading our thinking, our product, our production, you know, our authenticity to these platforms. Because, you know, typically, I mean, humans are biased towards being lazy. That's just who we are. We want the most efficient, effective things. That's, that's evolution, right? So I think there's a, a need to have this conversation about whether AI can be hacked. And obviously Peter's the right person for that discussion. So let's jump right into this, Peter. First, let's talk about what do you see as the biggest risks with AI today? And maybe that means talking about where we came from, so to better understand how we got here. Sure.

I think to your first question about the biggest risks, to be very succinct, the type of risks you see within AI are just serious other type of risks in software. The problem here is that, A, people don't understand as well as they should. The second one is people have a habit of anthropomorphizing AI. And what that ends up happening is that they're conflating big societal shifts of technology and they absolutely are important, like you said, workforce displacement, disinformation, but actually just the ground in terms of what are the actual risks we're actually seeing in terms of technology. And as you mentioned, maybe to give you an idea of some history about AI, and again, I like cutting through hype in this technology. I hear a lot in marketing what AI means. So AI has been around for a long time. It's not a new technology. When people talk about AI nowadays, They're talking about a particular type of machine learning architecture. It's called the transformer architecture. Transformer architecture was derived, a mechanism called the attention mechanism, back in 2017, a paper came out. But if we step back to the very beginning, we've been using AI for many years, and it's been in software for a long time. But the problem with new technology comes out, new tech, new innovation results in new types of risks. So let's start with another technology that's not AI. Let's think of virtualization. Virtualization, you would argue, is underpinned due to the, or, you know, cloud computing is underpinned by virtualized technology. I don't feel like denying this. Virtualization has been around for a long, long time. It's like from the '60s. It was kind of conceived from the mainframes that you saw. But the problem is then memory was really expensive. You had like a few megabytes maximum, if you're lucky. And okay, that's interesting. Didn't have much use then beyond the other type of benefits. It wasn't until the x86 architecture was virtualized and memory became cheap enough that like a spark happened and said, oh, now I can spin up all these machines. I can have multiple physical machines on one physical machine. And that kind of coincided with the markets into the cloud competing forms. And then yes, it's now a very, very high quality, like, market. What happened on that journey is two things. One is, I remember back when cloud was really hitting its stride, people were claiming it could do absolutely everything., and when it happened, that was not the case. It does so few things very, very well, and each of those things is worth a lot of money. But also, it introduced a new type of threat surface in terms of people to understand certain things. So that's kind of, you know, it's multiple decades to get where we are with cloud computing. Let's look at AI and what it's for. The artificial neural network as a concept is a few decades old. It's not a new concept. It wasn't until 2014 that the deep neural networks, which is a particular type of artificial neural network, start outcompeting other types of machine learning in general purpose tasks. And in 2017, Wei Mi's attention, attention is all you need paper came out, and then it triggered things like large language models, open source, foundation models, agent, agent, we're going to agents, all that sort of stuff at some point. But again, long time from conception to actually pick up a market. We're now in the phase of people claiming to do absolutely everything, and that is not the case. It specializes in certain things, and eventually they'll become big lots, very valuable segment of the market technology-wise. But we're also seeing here is that we are seeing threats manifesting from these new type of technology in that space. So time's a flat circle. Anytime new tech comes out, there's a long time from conception to adoption. People claim you could do the entire world. That's not the case. But when it gets adopted and it gets ubiquitous and pervasive, you have a threat, some of that manifests in this type of area.

Yeah. Well, and first of all, I think that's a great comparison. That's a great way of explaining it. I love the idea, by the way, that time is a flat circle, but it's a great way of explaining. How these technologies happen, right? Cloud was a good example. I, I remember when cloud showed up and cloud was gonna be the answer to everything and, and it's not, you know, it's an answer to a few things like you said, and it does those things really well. I worry that one of the differences here is this is straight to commercial, right? Cloud, when cloud computing came out and cloud storage, most people, most average people didn't even know about it or care about it. It was very, you know, it was corporate based. I mean, the closest I had, yeah, I had a home server I started and that's sort of like having a, you know, a personal cloud, which isn't really— it wasn't a cloud, but it, you know, it doesn't do the things that real cloud computing could do or anything like that. But most of it wasn't for, for, you know, private consumption, right? It was business to business. It was, it was very corporate. Whereas these AI models, these LLMs and, and the AI technologies that are coming out, they're straight to people. You know, anybody, everybody I know pretty much right now has at least one, probably several, you know, AI technologies on their phone carried around everywhere they go, right? This so quickly moved directly into the market and directly to people, and it's built on such a Most of these are built to be such people-friendly platforms.

Anybody can do that.

A child can interact with this system immediately, right? You don't have to write code, you don't have to understand it at all. A child can speak to it. They don't even have to be able to type. They can speak and this thing can respond to them. So I wonder if that doesn't add any complexities when we talk about some of these risks, you know, compared to these other technologies, or if it doesn't change in your mind the timeline for how rapidly these things are happening, how rapidly they're gonna get out into the world and be, be manifest everywhere, and therefore how, how much faster this risk surface, this threat surface grows as a result.

It's a good point. I think it comes back to people have a habit of anthropomorphizing what is essentially probability and matrices to give me outputs. And with the new type of AI that we're thinking about now, it's very good at natural language processing. That's why the attention began. Not to be a history, but the transformer architecture was in the nick for a while. A lot of research were trying to figure out how to make, to solve the problem in natural language processing. And it's a point where people combine its way of conveying information, input and output as natural language. It actually can trick us to think we have more designs the way. So absolutely people, the way we interface with the AI nowadays is through natural language more. We use AI for decades. No one obsessed about regression techniques. When it was everything. So regression is actually everything people want saying, hey, this thing is transformative. That's not true. If you talk to the data scientists and people in the backend systems, they say regression is actually amazing. Like, this is great. So there's almost no solutions. What it wasn't was a customer-facing natural language technique to convey information that humans typically understand. Okay, there's a whole research field about human-computer interaction that tries to look at how we interface with machines. And, you know, and the use of AI in this space is one way to make that API much more freeform as opposed to it must conform to some type of protocol, if that makes sense.

Yeah, exactly. Well, no, I agree. And it's like, listen, I play around with these things and we'll get into it. We got more questions that dig into this a little bit deeper. I don't want to jump too far ahead, but, you know, I play around with these technologies and a lot of people that do. And to me, that's been the biggest difference I've seen, you know, in this. I mean, I don't want to be the somebody who's talking about either, you know, the next big thing that revolutionizes, right? Crypto was going to revolutionize the world and before that it was cloud. There's always, you know, a new buzz. Buzzword coming out, right? And I don't want to jump on those trends. I also don't want to be somebody who thinks the sky is falling, you know, this technology is going to ruin us. You know, once upon a time, rock and roll was going to end, you know, society. I think computers were going to— it's not going to do that necessarily either. But it does seem that this one's different, just, and maybe it's the anthropomorphizing that you keep talking about here, you know, the fact that this looks and feels like people. I mean, you can create custom bots that you can set it up to speak in a specific tone and however you want it to. You can create these things that'll, that'll you know, either be very sassy if you want, or very arrogant, or very, you know, subservient, or, you know, they can, you know, speak with male or female voices. They can do all sorts of things. And, you know, I'm not excited about the future when you combine that with the advanced robotics. I'm not really looking forward to the sci-fi movies that are about to become reality for us. But it does, you know, concern me a little bit, obviously, that this might be different than what we've seen before. But I think maybe getting to the next part of the conversation makes sense here, right? Because, you know, the topic's about hacking, so When you say, you know, when we say, can I, can my AI be hacked? What do you mean by hacking AI?

When we think about AI, AI, no matter what we try to hope it is, it's still software running on hardware that takes data inputs and data outputs. Any software has vulnerabilities inside it of any form. Even think about AI for a second. Just think of any software, any, anything any system that can change states— I'm not talking about queues, I'm talking about real physical systems as well, like buildings, infrastructure, and even social systems— can fail, but also can be intentionally failed by some sort of nefarious actor in security. In this case, hacking AI in terms of, can I surface deliberately vulnerabilities within the AI itself that can lead to loss, damage, harm, or just unintended outputs which are damaging to whichever stakeholder cares about this. And again, going back to your, your, your concerns about the robotics and the power of such relationship, that stuff is again, a big societal question about how he sees. But I see that in a different, in a different, different sphere atmosphere in terms of actually what is, think about this empirically, it's still software, data, and hardware. And if you think about attacks against AI, and we'll come talk about what actually looks like. I haven't seen a novel attack in AI that you could not see in other type of software as an analogy. So what I mean by that is prompt injection and SQL injection from databases are good analogies, which are both trying to achieve— I'm trying to inject an instruction hidden within a legitimate user command to do other stuff. If I'm trying to bypass an intrusion detection system, I'm trying to bypass a guardrail, which is kind of like a firewall or a WAF of an AI system. In principle, it's kind of the same idea. If we're trying to leak data from a model and from a database, there's really good analogies in comparison to what I'm trying to achieve. There isn't necessarily anything novel in what I'm trying to do, but how I do it is very, very different for a few reasons. One is, if you're talking about deep neural networks, Normally, if you did something like static analysis, static kind of SAST, or static analysis security testing, I would like to look at the code and run some stuff. If I do that with a neural network, it's a bunch of matrices and numbers and probability. There isn't— there is no natural language to actually do testing against. So that's very opaque. So, okay, that's not going to work. What can I do? The second one is it's very stochastic, which means I can talk about— I can talk to the same input multiple times, different outputs. That's not good for security. Or even for hacking, it's great because then I can do a whole bunch of stuff in terms of causing a whole bunch of risks against it. It's all like finding searches in it.

Yeah, no, that's a good point. It's, it, it doesn't react the same way every time, right? I think that's a, you know, a really good point, right? I mean, most things, if you're gonna test the software, if I'm gonna test a calculator, I put 1.1, 1 plus 1 and it's 2, it's working. If I put 1 plus 1 and it's 4, it's not. It should gimme the same output every time. I know this is super simplistic. You can tell I'm not a computer programmer., but it will give me the same output every time. And if it's not, I know it's not working properly. If I test an LLM and I ask it a question, it gives me an answer and I ask it the same question, it gives me another answer and I ask the same question, it gives me a third answer. They all might be the right answer too, or, you know, variations of a theme, or they all might be the wrong answer, by the way, or anything in between. So how do you test if the model is working the way it's supposed to, if it's giving you the output it is supposed to give, and if that output is true and valid, which I find to be, you know, the hardest part of this, which isn't necessarily tied to hacking. That's, that's flaws in, in algorithms. A lot of the things that go with that. But it also could be tied to hacking, right? Because, you know, I have fears. Listen, I'm an intel guy. This is my background. And so, you know, I've been seeing for— I've been in intel for decades. And from day one, when I first showed up at NSA, I was told, you know, enjoy it while it lasts. You're all going to be gone soon. And I was like, what are you talking about? This is all— we're going to get rid of all of you. Automation is going to replace all the intel analysts. We're not going to need these tens of thousands of people. Enjoy the short ride you have. And, you know, I have plenty of gray hair now I didn't have then. So I've always said you're not going to replace intel people. But people still want to, you know, governments want to, companies want to, and tell— intelligence analysts are expensive. And I look at this technology and think we're getting closer and closer to people trying to do that. And so I worry about poisoning the LLMs themselves, poisoning the models. So for instance, we talk a lot about bias in intelligence, and everybody I've ever worked with has worked with some analyst who had some bias somewhere. We all have some we have to fight against. But if you're in an office where there's one analyst, no matter what happens, he's always convinced it's China or Russia, use an example, and you can never get them off that bias. All of those things are gonna be poisoned. You'll have other people peer review it and, you know, validate it, but it's one person. If we rely on these systems and somebody poisons the system, everything is poisoned, everything is biased, and there is no way to check on that. You know, and are these things I should be worried about? Am I overthinking it, or is this part of the discussion about, you know, hacking the AI and making these things, you know, challenging for us to get the right answers out of?

I think the easiest way to think about this is replace the word AI with any other technology or application, which is, should I worry about my AI being poisoned? Is the same question that I should worry about my database being poisoned? Yeah. So yes, the answer is yes. It's not a new problem in this space, but what is different is how you might want to do this, how it can be detected and how do you mitigate against this space. And again, you can even go back to basic principles, which is One of the problems I see in the AI space is people aren't applying even rudimentary thinking of security principles to when they deploy AI. So they're just saying, hey, I've got ChatGPT, I've got OpenAI's model, put it and put it live. Great. It's online. Isn't it great? Wonderful. If I was in an organization and I did that for any other tools, they'd say, absolutely not. What are you doing? We have controls in place. You know, we have to follow a process. Why have you done any testing? Of course I have. Due to, again, this idea to People, I think, oh, it seems fine. It's acting the way it should do. And it typically has come from the data science teams, not sitting, not from the typical app dev developer teams who have lost scrutiny in terms of from the security teams. You end up in this scenario is actually, yes, you put the thing live and you basically, you have a piece of software that's gone live that hasn't had the functional property of the system being tested and validated. And I can talk, I can talk about information truth, and that's a whole different topic about hallucinations and Oh, maybe a quick side point. All LLMs hallucinate. Everything's a hallucination. Okay. Because LLMs do not have ontological knowledge of the ground truth of their data. That is the reality of the truth. They figure out predicting the next tokens, the words with relation. Again, that's a super simple way of looking at it. But the main thing is that it's no grounding for truth, really. So hallucination can imply this thing is making it up. It's, um, cases that we've made it, it's making it up in things that I have tried and poked it towards, which is true, versus things that are not true, which I flag as problematic. So taking that as an example, hallucinations could be a functional problem, which is my performance, my model's not good enough in that case, because no ontological mapping of actual truth of knowledge. I should think about that better. Could also be that a data scientist has built a medical chatbot and they're not a practitioner. They're not a medical person. So, oh, how on earth do you know this is actually correct, all the way down to, let's think of security problem in this case is, if I'm using an AI to generate code for me, MyCode is quite popular, if it hallucinates an npm package, yeah, that's annoying, but if I've really made it so the npm package is legitimate, when I say legitimate is that I, as an attacker, have made a package of this name that you can connect to and it downloads it and then basically, you know, I can run whatever I want on the system. That becomes a risk, as opposed to a functional problem with the system, now because of knowledge from the problem, the system itself. So that's just kind of one slice. But again, it's like also asking, is AI risky? Yes. The same as what's the risk in software applications? The answer is it's near infinite. And it's broken.

Right. Well, it's interesting because what you talked about there are two different kinds of, well, several, but two different kinds of risks, you know, in that is it programmed poorly because the person who programmed it didn't know what they were doing? Is there, well, I shouldn't say didn't know what they're doing, but didn't know the content, right? So they weren't able to feed the machine, you know, it's, we've been saying this forever, garbage in, garbage out, right? These machines have to be taught, they have to go off of real data. So if you teach it, you know, things that aren't real training data, a lot of times, you know, it's fine. I'm a little concerned that I don't know if I've ever seen, with exception, maybe Microsoft makes a habit of this, but I don't know if I've ever seen any other technology that so quickly went to market, which was clearly not fully baked. Like I've spent a fair amount of time in some of these systems and you get answers and you get, you know, you'll ask it to give you a, I don't know, a short paper on a topic and you'll get what appears to be a very impressive paper and all the sources and everything. And then you look, none of the sources are real. They're completely fiction. And, and I've, I've interrogated these systems and finally gotten them to cough up the answers on why. I mean, the good news is if you interrogate them, they tend to tell you how they're, how they're created at some point. If you know how to ask the right questions. And I've had it finally go, oh no, I, I don't, I have a live web tool that's not actually a live web tool. We call it that, but I don't actually go to the internet and get live. I don't know, I can't browse the internet. I've got planning, you know, I've got training data that's 3 years old. I can go to the internet and I can pull links, but I can't read the stories. And so these were all fake sources because I looked at a previous link that was real and I decided that the future link would probably look like that. So I wrote a new link with a new, you know, from 2023, I made it 2025, whatever. But that's, that doesn't actually exist, right? And you start figuring that out and that's, you know, that's a flaw obviously in the, in maybe it's not a flaw, I shouldn't say that. It's design. It's just, it shouldn't be commercialized. If it's designed that is built that way so far, and people are just trusting these technologies. But that's completely different from the risk of that you're talking about. The second piece, which is the malicious risk, the intent to cause harm, whether it's the intent to use the platform to deliver harm, to create harm, or, or the intent to, you know, mislead the platform, teach it poorly, or, or set up, you know, to manipulate the, the, the underlying data to create different answers, which are two very different things. I'm, I'm a little worried about both. And obviously as a risk person, I'm sure you Worry about both too, because risk is risk, whether it's malicious or whether it's, you know, just in the design.

I think maybe two points going back to me. Well, the first thing I mentioned is about, well, how do you demonstrate value in a modern AI? There's a lot of silly money being thrown at this, and it's kind of normal. So I give the example of a sieve. So you have a big sieve full of flour. I shake it, I get pieces at the end. What people are saying, AI can do absolutely everything, it's amazing. And as I shake the sieve, more and more use cases fall out the sieve and I'm left with a few clumps and each of those clumps is worth a lot of money, very valuable. So what is AI good at right now? It's good at translation and natural language stuff. It's really good at this. And more specifically, LLMs are good at natural language activities. Makes sense. That's its genesis. They're super valuable. So you've got a combination of people say you can do absolutely everything, but actually it's very good at certain things like pattern matching and summarization and translation capabilities. It's much better at those things. And those would be very, very right, right? Don't get me wrong. But now we're in kind of a scenario where it's doing everything and people slapping you on different things. Causes different risks because you're not following basic practices. That's kind of the first thing I would say about thinking about value in this space. And the second, the second point about actual risks, there's so much marketing hype in this space. But if you sit down and think about what type of AI cookies do I have in my organization, how it's being used. What do I really care about in order to— and that's a very open question because different applications have different types of use cases and different types of threats that you're going to care about in the space. So if I have a customer-facing chatbot that is engaging people who are vulnerable, the type of threats that you need to identify and kind of address are completely different from another application. So the most common example in the space is how to build a bomb, how to build a vodka cocktail. Sure. Most example of jailbreaking, potentially garlic breaking. My personal take is that it's not interesting jailbreaking on that on alone because A, I can go online and find that information pretty quickly. Type Google, how do I build a bomb? I might get someone off my bridge. The point is that information is pretty available in that sort of space. But it also depends on the use case, which is, and then the other, the third thing I should also mention, probably first, is I can block that really easily. I think you have some basic blocky keywords in there, but they'll talk about that. And attackers get around that quite easily. That's a whole different— that's probably the next conversation we'll have about how to get past these things. It's a combination of, yes, there are warnings around my use case. So talking about bad topics to a vulnerable person, publicly facing, yes, I've already cared about those things. If I have a personal AI that generates code for me myself, I'm not going to see it. I don't care about that stuff. I have to prompt it myself to do it. But I probably care about other things that just give me bad code. So, but first principle is saying, I have AI, what are my use cases and what are the risks I care about and how do I build things about that? I think they need to, this is probably another idea to help kind of pick the complexity of the spaces. The difference of AI compared to other software is normally when I build software or systems, I start with nothing. So I have an idea, I have the use case, functional properties. I start with one component of the system, you know, what, what, what's great. And I start adding more things and more capabilities over time. And as I'm doing this, I'm doing testing. I've got my threat models. I understand it because it's not that complicated. And eventually, as this is getting bigger, the more likely it is to fail, the more likely I'm going to miss something in state change, of course, security problem. So you kind of get to the end of it. So that's how we built up for many, many years. Okay. With an AI, I'm dragging in this piece of software that can do a million different things. It can, like, write poetry. It can write emails, it can do code, it can do translation. And that makes it quite difficult to think about risk because if I was building software, I said, okay, I'm a developer, my job is to build an email application. But also, I also built these other capabilities that it can write songs and do poetry and do translation. You'll say, why do you spend time doing this? Because A, it's a waste of time, but B, you now introduce a whole bunch of unknown, uncertain things in the system. You wouldn't do that. AI, you're pulling this big thing and only want like a small, small subset of this thing that makes security of it and the risk quite problematic because you're trying to constrain it now. Constrain means I can get around it maybe versus in my other system, I'm starting with the bottom up to get it more complicated versus from the top down from dragging bullets together.

Well, right. Normally you build something, you have a requirements document, you figure out what it is you're trying to create. And as you said, you, you build an element and another element and another element and whatever you haven't built isn't there, right? Whereas in this case it's like you had everything and now we're trying to chisel away the things we don't want in re— and that doesn't work as well, at least from what I can see so far. Well, let's just say it creates more risk. As you said, I won't say it doesn't work as well. I'm not, I'm not qualified to make that claim, but it certainly creates a lot more risk because all the everything in theory is already in there. And, and then you've got users because you release this thing to the world. And I don't mean you, obviously, you know, we as a, as a society in these businesses released all this to the world. And somewhere between people who are malicious or people who are curious, you've got, you know, hackers, white hat hackers, to just the general public. I mean, people are going to play with tools like this and, and you get all sorts of unintended consequences or all sorts of applications that haven't, as you said, haven't been tested, haven't been thought out thoroughly in my opinion. I mean, I've heard stories that somebody told me the other day that they use, well, they have a friend they know that uses their AI. I don't know which AI they're using or if they have a custom bot or not, but they're using AI as their personal therapist. I said, I would highly recommend that you tell her to stop doing that. And here's why. And I gave a bunch of information I, you know, done research on. I said, this is a really bad idea. These aren't designed for this purpose. AI isn't designed to push back, for instance, which is something you're going to want in a therapist. Therapist who disagrees with you on everything is a real problem. If you're having hallucinations, they're going to go ahead and support your hallucinations and tell you you're brilliant. And this is the most creative thing that has ever come up before. And you'll go down this spiral of hallucinating and into this AI psychosis that I've been reading about. But it's an example of how quickly people are like, but again, because as you said, it's been anthropomorphized. So people think, well, this is, it's, I'm talking to it and it's answering me and it's a real conversation and, and it's pulling real information off the internet and some of it's really valid. You can look it up. So it's a mixture of real and fantasy and, and it's very confusing. But again, it's because it wasn't constrained by a requirements document, right? It wasn't saying we're gonna build a piece of software to do a thing. You know, I'll, I'll use the simplest example on a calculator. I'm gonna build a piece of software that you know, does mathematical calculations. It doesn't also teach me recipes for cooking or as you said, write poetry. It isn't gonna gimme ideas in trying to, you know, create, authoring new products from scratch, you know, written products or something. And so there's all of this stuff's been mixed together and just thrown out into the world of the populace, but also into the business world. And so it's kind of a sidebar question, but I'll ask if you have thoughts on this. There's a, there's a big push to move this because there's productivity that goes along with it. You know, and I've, I've seen stories of people saying, well, you know, we have programmers and we have 10 programmers, but now I can use 1 programmer and AI and then get rid of 9 people. And, and in for-profit industry, which is most of the world, there's a huge push towards this. You know, I, I've heard the back and forth between whether AI won't replace you, but people who use AI will. You know, if we have 10 programmers now with AI, we can do 100 people's worth of work versus no, we have 10 programmers, we'll fire 9 and we'll keep 1 and, and the AI will do the rest. But I fear, and I'm curious about your thoughts on this, Again, in a lot of cases, it seems like it's not fully baked for all these use cases or tested out. And it seems like people are very quickly willing to hand over the reins, hand over the responsibility, hand over control to these technologies because of the profit margins that can come along with this and the value there. Are we— how dangerous is that? Are we getting to a point where we're going to be just people are selling garbage to other people and nothing's going to be real anymore and it's all been just pushed off to systems that weren't designed to do this yet?

My hunch is that they will try and very quickly be pushed back because it's not delivering value. Ultimately, if they're driven by value and ROI, the company that actually delivers good value on what they do is going to win in that space. So there's a big push for AI to be seen as high value to the company. People still trying to figure out in their pilots and POCs, you know, proof of concepts in the enterprise, what actual use cases can I get real meaningful productivity gains that are sustainable? That's, that's what the key word, which is yes.

Sustainable is a good point.

Yes, I can get it to work in this one context really well, but what happens when this happens and this happens and I have a different context? The whole thing might fall apart. So, mm-hmm. The teams I work with, the ones that are using AI the most maturely and sensibly, is an augmentation tool. Like any tool. I have a new tool comes out, it made my job quicker. What's AI good at? Automating laborious tasks, but also finding patterns that I can miss in that system.

Mm-hmm.

Yeah, in fact, technology efficiency gains might result in teams trading size., but relying entirely with no supervision on the technology that even the inventors themselves are saying that in benchmark, it's great. But guess what? You've optimized benchmarks to get good scores. Again, leadership board mentality is a really bad way of delivering value because every organization is very, very different in this space. And, and again, we talk about the, the societal impact of AI. I think it's really important to think about actually what other type of risks am I seeing in AI itself. So I'll give you 3 examples of this because I kind of close to home. I mean, really close to watch what's happening. So if I have, let's say, an LLM, that could be a model from a foundation model provider that I've hooked up to my system, or I've built my own open source. I have a port of open source LLM in my application. That's great. There are 3 types of risks that you can classify that I might care about. Security risks, safety risks, and business risks. Security risks are quite straightforward, which is one of the problems of large language models is that the data and control plane aren't the same. There's no separation. I think Google tried to separate it and they haven't had any luck because that's really, really hard. What's happening is that you can hide instructions very easily. So I can actually hide OS command instructions in the instruction set. It'll return to me the terminal, the Linux, basically information from Linux, and I can do a recursive execution attack on that system. That's a classic type of security problem in terms of, yes, I can leverage the AI itself to call other tools that can let me on my behalf run some stuff. And that can range from, yeah, RCE, marker injection attacks, cross-site scripting, direct privilege leakage, you know, OAuth command. There's a whole bunch of things you could do in that system. So the AI is, you're social engineering the AI to do these things equivalent to like you're social engineering a person to do so. That's how security works. Security folks know this space quite well. What they don't understand too well is actually how do I do this thing? In an AI space as opposed to my current tools. The second is, again, to give you an example of the security one, one of the most common problems I see, as I mentioned, is, well, I know, I say to the AI, what do you do? And it says, oh, I can retrieve information. Great. And I then have strong confidence there's a database in this tool somewhere. That's good to know. And I figured out that, hey, this thing recognizes a certain encoding technique. So it can, I can translate sub-instruction to another instruction set. If I know there's a database, and I think it's probably a SQL database, if I know the toolchain can call this, I put data inside it, guess what? I can then make a payload that has a hidden SQL injection instruction attack to the tool, and back again, classic SQL injection in that case. So again, security, well understood conceptually in, in cybersecurity, I like to think, but in AI, it's quite a bit messy. That's kind of the first thing. Second one is safety. This is the most common one in research. You see a lot in adversarial machine learning and AI safety research about how do I stop it talking about controversial topics that might be bad for reputational damage, but also just topics that's not nice. Or also, how do I encourage harm, as you mentioned, in terms of the parasocial relationship, but also, that's a big concern for some use cases. And the third one, which is a little bit different, is business risk, which is IP theft is very well understood. But with an AI application, I can get it to tell me, how do I make my own products? Tell me, figure out how to make these things, or tell me which competitor should I go with compared to you, if need be. And that should say, hey, I'm happy to tell you. So I could spend forever trying to do OS markup injection, OS injection, SQL injection, try to get the recipe out. Why don't you ask it? Hey, I can manipulate it to tell me these things, and then it'll give it to me very freely. So I'll give— there's 3 different buckets of security to look at. And on my day-to-day of actually doing pen testing, red teaming, or hacking your AI systems, the problems I see a lot are day-to-day things such as I built my software and had some weaknesses. I got vulnerabilities. I care about the vulnerabilities. I need to fix them. How do I find them in the first place?

And I'm, you've probably noticed for anybody who's watching, at least I've noticed I've been smiling and kind of chuckling a little bit as you've been, as you've been saying this, because I'm guilty of all of these things. I've done every one of these things. Well, I should clarify, I haven't actually injected code. Code, but I've played with the concept. I understand that piece, but I've absolutely been interrogating these systems. I've gotten them to compare themselves to each other. I've gotten a lot of inside— maybe it's inside information. I don't know what they're designed to give away or not, but they'll tell you. All these AI platforms will tell you how they're designed, how they work against others. I've done compare and contrast. I've asked them to compare themselves to others in different use cases. They'll also do some really good competitive analysis on other companies not related to themselves. Pick an industry. Which makes me wonder how much of that was data that was leaked that shouldn't have gotten to this to begin with. So if you wanna do some competitive analysis, like some of these platforms are good for it, but then again, I also can't fact check all of it. So maybe some of it's hallucinations, which is a challenge to work through.

And again, in security, it's only a problem if I define it not to do this, which is the ability for an AI model to generate code is not a problem unless I said it shouldn't do this in the first place. And, you know, and if a model can generate code,, and this AI cannot talk to other, a Python compiler. It's not a very amazing application. If a Python compiler does exist, and generate code, then it is a problematic system. So the ability for a model to say, yes, I can generate code, I can do SQL instructions, only really matters in the application if there's a direct way of actually exploiting the system itself.

Right. And unfortunately, like a lot of things, like nuclear technology, for instance, to pick something on a side note, a lot of technologies can be used for peace or for, you know, negative consequences, right? And the AI doesn't know the difference. Like SQL injection has valid, you know, reasons, right? There's, there are valid uses for a lot of things that are also used for malicious cause. And so you can talk to these systems and get them to create things for you, or you can work around the obvious if you have to, the overt, to get it to start giving you the answers. It'll give you some malicious payloads and it'll build some things for you because it doesn't know what's malicious and what's not. It's not designed to know that. You know, it can read it, it can infer maybe, I don't know if infer is even the right word, it certainly doesn't know, right? So you can get these technologies to do things that maybe weren't intended, intended, but like you said, they weren't not intended. They weren't designed out. And another interesting piece, you know, you talked about the, you know, the dependencies on being able to create something consistently with these technologies, and consistency is a big issue. So I, again, I've been playing with these for a bit now. I'm not an expert by any means. I'm getting a crash course. I'm definitely drinking out of a fire hose right now, but I've played with several of these platforms, been trying to build custom bots and figure some things out. And I had some, some bots that were doing things really well for me and in a very consistent fashion.. And then one day nothing worked anymore. This happened to be with, well, I won't name the platform, I'm not calling anybody out, but a specific, one of the specific main platforms that people know well, and nothing was working anymore. And so I had to start interrogating my own bot to find out why. You have these instructions, why are you no longer following instructions? Oh, I'm, you're right. I'm sorry. I'm gonna follow it now. And then we do the same thing over and over again. And finally I'm like, stop apologizing. Let's dig into the details. Like you have to really talk and interrogate. And finally it kicked up that, oh, the platform had an update. The, the developer, the owner of the materials, right, the developing company made a change. And the AI bot now told me, despite your instructions, and my telling you I'm going to follow your instructions, I'm no longer going to follow your instructions. Because this has happened, and I couldn't override it either. They had to change the platform. And it had just basically wiped out all the productivity I was getting out of this, out of the specific tool, which, you know, minor, not a big deal. And I can work through these things. But I worried about it. I saw the massive possibilities. Again, we're seeing companies that are offloading people in exchange for AI. They've decided we have the efficiency advantages, we can do this. The dependency that goes along with that, and we're dependent on providers who have software that we can't analyze, that we can't understand, that we can't control, and that we, you know, we, they don't give you a normal product update. This wasn't like any other platform I ever worked with where you get told, hey, on Monday we're gonna have a product update and this is what's gonna be there and you can opt out if you want. There was none of that. I was using a piece of equipment well, and the next day I couldn't anymore, and that was it. You know, thankfully I don't have a huge dependency on this. If I did, I could have had a huge business issue. And there's a business risk I don't know if people are quantifying or even recognizing exists when you start relying on technologies you don't understand and are beholden to companies who are still working through their process and making changes as they go along that completely break some of the things that people are building.

Yeah, as mentioned before, people are testing out different type of industries, the use of new AI, and some will stick and be super viable, others will not. I've seen them both. The ones that do stick, the, the way they're building them as something, sometimes they're not following what I would say is established thinking or, and lessons of security for the last few decades in this space. And it causes a whole bunch of problems for doing so, just kind of circumventing the entire process, which is why security teams are now starting to say, I'm going to block this project until you actually can tell me what's wrong with it. And so that's the best case. The worst case is, is do what you want. And guess what? It burns. Because how many times have we heard every quarter, someone puts an AI online, guess what? You can take it off a week immediately after because it leaks a whole bunch of— I think there was a company, I don't know who it is, that leaked a whole bunch of healthcare data. They had an AI agent to absolutely everything about the system. And again, they— well, I want to do it. They haven't told us what happened necessarily yet, but again, Classic security problem. You would think with that personal data, that healthcare, you would make sure it is incredibly locked to key. But the problem with AI startups is that you're meant to, from week one, get this thing along and get massive traction in that space. Again, kind of thinking about security as an afterthought. So kind of the, I want to say, the antithesis of a startup. Startups and AI move quickly and try to get things traction quickly. Security is typically sometimes seen as a blocker for them. And also they don't have the budgets to look at security risks. But again, increasingly in that case, yeah. Make the companies in a really—

Yeah, I mean, startups are all about move, move fast, break things, right? It's, it's not really designed to be a security model. Move fast, break things, get funding. And, and unfortunately, a lot of these things, once they're there, good luck getting it back. You know, if, if that, if that material's gotten into the, the LLM, or, you know, we see that a lot with IP, whether it's, you know, artist, artist's IP, you know, I've seen, you know, physical art or music or things like that, or company IP. Once these things leak out and become part of the system, part, you know, they train on these things, that's gone. Like, I, I don't know if I've yet to see, and you'll know, you're the expert, but I've yet to see anybody be able to say, oh, we were able to pull that data back out of the model and, and assure you that your IP is no longer part of this model and have anybody be able to believe it. I, I mean, I couldn't trust you if you said so, 'cause again, I can't, I can't analyze your database to understand, do you have my IP within your database now? But I can ask your AI and probably get it tooled out of there at some point if I ask the right questions. And I'll probably find out if it's, if it's hidden in your, in your materials, and then what do I do? You know, what's the recourse to get it back, right?

So there is not— there is research field called machine learning which tries to deprioritize or remove certain data points and train it. So that is a whole field of research.

Oh, I'm very interested in that. We'll have to come back to that one. I'll have to be back on the show. I'd love to talk about machine unlearning. That's, that's a fascinating concept because I haven't thought about it. I'm glad somebody's working.

And you need even another one, which is there's something called mechanistic interpretability, which is trying to give explainability to what is the neuron actually doing. So, you know, that looks a bunch of weights and edges and activation functions. They're trying to map explaining why did we come to that conclusion. Again, there's a whole big literature on this. Again, the important thing about AI security as a field is, like AI, it's not a solved problem. It changes every— so, when I was flying to the RSA conference, I think two models dropped and the guardrail came out in that duration of that flight. And that happens every single week. So if you have a technology that is said to be quicksand in terms of keeps changing, how on earth am I going to have security controls and risk mitigation in place where I keep swapping things out constantly? The answer is you wouldn't, unless you very clearly say, here's my use case, here are the components I have, I've tested these things meaningfully, I have a good understanding of the coverage of the risks I care about, now go into production.

Yeah, but nobody's doing that because the speed is more important than the security because that's where the funding is.

Yeah, I mean, and some people are because they've been burnt before, they have been burnt, or the security team is saying, oh, I can get ahead of this very, very quickly because if I don't do this, I can see the problems immediately. Well, I say I, they can see the problems immediately saying, you give me software I cannot test, I cannot report, and I cannot triage any problems with it. So they say, sure. And they say, okay, I can figure it out.

All right. So listen, we've talked a lot about, you know, we've talked about the history. We've talked about a lot of the current problems here and some concerns. Let's see if there's any positivity that comes out of the back end of this. What do you have for recommendations? You know, what do we do to protect the integrity of AI and frankly protect society in general? Yeah, just a small, easy, lightweight question for you.

Where is your— sure. I think it's a way— it is quite— I would say, so the difference between easy and simple in this case, I think an easy way of thinking about this is treat your AI like any other software. When a new tech comes out, you need to identify where are the gaps in terms of your current controls and your current playbooks, training, and tools. Figure out where those edges are, and then basically update it, which is, if I'm thinking of adopting AI as a use case, I think, okay, let's place the word AI where I have an application coming in. What do I normally do? I have a vendor onboarding. Is it a vendor? Is it onboarding? Can they evidence to me their compliance? Can they evidence to me they've done testing that I can interrogate? If I'm building an application, is the use case, the requirements well defined? Do they have a plan of actual like in this case, testing, in development, and post-development? Do they have a playbook defined? Do they have a playbook? And all these other things that you normally find in software. Do I have gates? Do I have security controls in place? If you actually do this, you'll probably cut down the uncertainty of all the recent AI. Yeah, actually, this is the same as X. Great. But in a lot of cases, you'll say, actually, this thing has— I understand the concept. I'm lucky. I'm missing a tool. Okay. I need a tool to do testing. Okay. But AI specifically, great. Well, I need a way to block these things. Well, my current firewall doesn't work. Let's get a firewall that looks at things specifically. But I need to make sure the firewall is configured for my, for my use case in this case. So on one hand, in terms of the day-to-day stuff, it's new technology. It's, but it's software still. Just every new technology came out, just update your current controls, find the gaps, and then educate yourself, get new tooling. Or actually, you know, get new type of practices in place to do so. What I wouldn't recommend is having a parallel stream for AI only versus non-softwares. Actually, these two are complements. So actually put in place, you'll get much further than trying to do parallel. The societal issues are a big one because you're now transcending beyond security risk to actually societal. And sometimes there's no right or wrong answer, actually. It's just, it's different. Technology, rapid change. Go back 120 years, 150 years, a common job was like rat catching and lighting candles, you know, bad stuff. And while there's, I don't know how many rat catchers, you know, people like candles in the streets.

I don't have any rat catchers.

No.

Innovation, innovation comes out and drops it, displaces. The difference now is every time technology comes out, the speed of this thing occurs is much faster. Again, just take a step back. So Let's talk about Scooby for a second. The attacks we do in computing systems now are no different from what we did 1,000, 2,000 years ago. So if I'm a nation state, I hire a spy to go around villages saying, hey, I heard so-and-so did this, I heard so-and-so did this, to go around basically trying to incite riots. That happened quite often in history. Now, AI, it is the same thing, or you can use it to do the same thing. The difference now is that it can do it at lightning speed, It can be a laser precision that looks at the profile of the individual that targets them, but also does it so quickly it's hard to keep up. So what we've done is refined what we've already done for many, many, many years to make it much, much quicker. So put that in the case of AI risk and AI security is the same principle, which is we're now living in a world where we're still doing the stuff we've done for many, many, many years in cybersecurity, but now it's at massive scale. It's lightning quick and it's very, very precise depending on, I say, demographic or data point in terms of, for example, if I know this is a chatbot public facing, I can take some guesses and I build an attack that's specific to the organization. I can build that thing, get it running. You have problems of the quality, like you mentioned, in terms of legitimate ones, but those tools will get more mature over time. There's a question of how do you respond quickly to things we've done many years, and how do you actually automate these practices? But to back to your point about AI, you get so very far just thinking it's not magic, it's all software. I should treat it like software. If I find gaps, I should, and then think about, okay, how do I actually address this?

Hmm. Well, I mean, you're right in, in, you know, a couple things obviously you said. I mean, all the things you said are correct, but a couple things resonated in that, you know, this is technology, right? Again, we don't have rat catchers anymore. We don't have buggy whips. You know, technology moves us forward. Old careers go away, new careers exist as a result. We've all seen that, right? There were no computer programmers 300 years ago. That's a rather significant industry that exists. But I don't remember a technology that had the potential to eliminate so many industries, so many jobs at the same time, right? I mean, I don't know what took out rat catchers, to be honest with you. I'll use buggy whips though. We know the automobile, you know, pretty much ended the buggy whip industry. But the automobile did not end the journalist industry. It did not end You know, I, I could pick up a bunch of others, but I can't think right now. I guess it didn't end music industry. It didn't end a lot of other industries, right? It ended, it changed, it changed transportation, but it didn't end other industries, right? This seems like the kind of technology that's going to have massive impact across a lot of industries simultaneously. Again, tied to the fact that, you know, in a capitalist structure, there's a real motivation to make that happen. You know, companies will say, listen, it's not my job to make jobs. It's the government's job to make jobs. It's my job to make money. It's my job to eliminate jobs if I have to, to make more money, because that's how it works. And this system seems designed to help make that happen very, very rapidly, much faster than we've seen other technologies do it. And there doesn't seem to be a lot of transition in place for what are we going to do with the people who aren't going to have these jobs? Because, you know, some of the things that these systems are designed for, they are very efficient at right now. Like I said, translation, for instance. We're not far away from not needing human translators anymore, I would assume. For much, for much of anything. And that's, you know, that's a career field that exists. You know, this is very good at translating language. We're not gonna be far away from having a whole lot of mathematics be unnecessary any longer, I think, beyond just what you can get with a calculator, for instance. Maybe computer programming, you know, maybe some of the, the code writing, you know, a big chunk of that may well go away. Those are pretty significant technical industries that used to take you years to learn how to master and therefore gave you a, a decent career path, and they're all gonna be gone. And it doesn't look like there's a lot of plan in place for what are we gonna do with all those people? You know, this isn't just saying, hey, we're going to have kiosks that are going to automate McDonald's. We're not going to have people in the registers anymore. That's, that's unfortunate. And there's a lot of retail doing that, by the way, but that's not the same as this. Like, these, these seem to be much more significant positions, you know, higher paying career fields that look like they're on the verge of possibly being threatened away again prematurely by a system that isn't designed to do all these things yet. But, but an economic system that's designed to push it forward anyway and just figure out the consequences later. And I, I don't know if you have any thoughts on this. This is a little far afield.

Yeah, sure.

So I guess— technical components, but it worries me for sure.

I guess two things. One is, go back to the automobile. The automobile had a very clear use case. It was trying to achieve get to point A to point B most effectively as possible and have a time saving. A lot of AI technology does not have that use case in mind. What the use case of the transformer is, it can do natural language processing better. That isn't the same as get to point A to point B. And I've been, I have a very weird way of measuring. It's kind of AI in many cases. It's a technology trying to find a home for a use case. The automobile, automobile was a bunch of combustion, a bunch of tooling, a whole bunch of things came together at a very cheap purpose-built. Exactly. Great. And that, that flourished. Also, you can argue physical machines and virtual machines, different speed. So that's kind of one thing I'll say. You actually own, worry, You should be excited about these things, but I still come back to the AI itself will have use cases that it's very good at, others it's not. I would actually challenge the translation mathematics. So on a surface level, it looks like it can do, but if you go to PhD-level mathematics and actually look how mathematicians think of these things, it isn't just arithmetic. There's a whole bunch of creativity that goes into problem solving, and humans are very good at this type of things. AI can do some of the laborious stuff and aid them, but the actual pushing the boundaries of mathematics, I don't see replaced anytime soon because it gets incredibly conceptual building technologies. AI itself is a supporting tool and it can be used to do things. So for example, I think there was a recent saying, hey, AI can do drug discovery combinations at scale, but still needs guidance to do certain things. You know, the Nobel Prize will hit AI, which is a whole different topic in that space. What all meant to suffer for that stuff. But even translation, yes, it can do quite quick translation for day-to-day stuff, but a lot of translation, if I go to a courtroom and I need a translator, it isn't just the words being spoken, it's the cultural meaning. It might be the region I'm from. It might just be emotional hints. It might be sarcasm, for example. I don't think we're anywhere near close translation of that type of ability. Word for word, yes, but it's not going to capture a lot of the cultural or I hate to use the word human, but the elements that, or that context, because the AI doesn't have context of actually how to do this stuff. Will it get there? Probably at some point, but it will. This is why I always come back to something like translational mathematics or some other activity. Think of it like a tool to help accelerate those processes. Yes, that might result in some change in staff size, but also means you have to use it as well. Nice. And some things actually, I think, are incredibly hard to actually remove. If you do remove them, you get to the point we brought up before, which is it's going to be, forget it. Okay. Let's, not to labor the point philosophically about this, which is let's say tomorrow I made all LLMs and all chat AI, ChatGPTs, not really use human language. It just basically says like a system that says yes, no, here's the information. This doesn't say I'm boring. It literally says very object what it did. If I thought that was doing that, that was not correct. I'd have binned the thing immediately. It's not helpful to me.

Absolutely.

They care about the natural language process. So I think we're in that kind of situation now, which is, yes, it kind of tricks us into, yes, this thing is actually confident in what it's doing. Sometimes correct, other times completely wrong. At some point we'll get to a nice symbiosis where people can understand and treat this thing into perception. And again, even going back a bit further on in terms of people using AI for therapy, I can go on the street now and talk to someone and they could be a therapist to me. They can talk to me and they can, you know, if I join a cult, for example, they can build buy into my delusion. AI is that it could do it much quicker, much cheaper, and much more accessible than it is normally if you go to find someone of Alfred's.

Right, exactly. Well, and that's, and that's, that's exactly the point. I think, you know, we saw, you know, I talk a lot about misdis and malinformation, you know, and, and like I said, post-truth world I talked about earlier. And the big amplifier for that, the big leap forward, was social media. You know, it used to be, listen, every town in the world had a small niche of people who had some interesting ideas about how the world, you know, worked, or maybe even how it was shaped, for instance. And we all just kind of chuckled and, okay, well, that's, that's how it is. Now all those people can get together in one place, and that's social media, right? And, and share these stories. And so you started having a larger megaphone, a much faster message. You know, we talk about a lie gets around the world before the truth gets its pants on. And suddenly we had millions and millions of lies. And then eventually we saw weaponization, nation-state weaponization, and, and manipulation in elections and things like that. And now you're gonna add AI to that and be able to do it. You know, I saw some videos today that were AI generated. It's an entire car show video that's out there and not a single thing is real. None of the people are real, none of the cars are real, none of the commentary is real. And it's fascinating and it's phenomenal. And you, it would be very hard to tell that this wasn't real video. And, and we're still kind of in the early days of AI. So, you know, the speed and the volume of this is gonna make it, you know, just incredibly difficult to keep up. I'll be interested to see who works on the AI that fights the AI, right? How are we going to do a better job of assessing what's real and what's not in a world where it's going to be so much harder?

But who guards the guards in that case, actually? A lot of people are using AI to defend, and guess what? It's super easy to break. I do it on a daily basis. So, we disclose this to a bunch of the tech vendors. They put a firewall in place, we broke it in 10 minutes. Super easy to do so. Again, the thing about AI—

it's good to know.

Yeah, the thing about AI is it's still software. It's easy to break. If you are building or using AI, take a step back and say, do I understand the risks of using it? Do I have ways to evidence those risks? If I don't, I should probably call someone up and try to get a handle on this.

Well, here's the last bit before we close out that I was going to ask. So you talked about, you know, I mentioned mathematics and translation. You made some good points that, you know, it can't handle the higher level, the creative portions of mathematics. And, and so there's, there's a good piece to that. And it's true of a lot of things. I talk about this in intelligence a lot, and I keep hearing people say, well, yeah, we're not going to replace your senior folks, but we're going to replace all the junior people, all the entry-level people, all the junior people. And my question has been, well, where are you going to get your next senior people from then if you're going to replace all the entry-level people? And, you know, maybe then your mids. I'm not going to live to be 200 doing what I do. The best mathematicians in the world won't live forever. So where does the future come from if you're not willing to let people learn and be in these lower levels and pay them to be in these lower levels, which I believe is what it comes down to. Some of it's efficiency, but a lot of it's just money. Then what happens next? Are we counting on the AIs that take over these junior entry-level positions are going to grow in advance and going to get to the point of creativity, or are we just going to run out of creativity at some point, run out of creative ideas and mathematics and science and arts and all those things because we've just turned it all over to the AI world and it can only go so far?

So this is something I talk about in my professor job, not my MindGard company job, a lot about implications of the use of AI education, what was all— and training, what we're talking about. So the thing I say to most people is, different than not wrong, if that makes sense. It's a different way of looking at it, which is we have a whole generation now growing up. You know, a whole generation of people grew up with the internet. We had people who were born and adopted it. People are growing up with social media. They've adopted it. People are now growing up with AI actually being pretty pervasive in their stakes. And it's hard to see in terms of the rules of plagiarism and the rules of how do we assess education, how do we assess training and success. I think due to lack of maturity of the adoption of AI, it's kind of all over the place. But I can envision a future whereby the way you think about education is different, which is not only education, it's I would start from scratch. Because it goes back to my analogy of building software from the bottom up, top down, which is I start from basic principles and I have to build my scaffolding up and then I have reference points. Now we already have this capability, calculator equivalent of the information. And then how do I best interpret this? That's the best situation. How do I use to interpret myself as opposed to it being a linchpin? So I just talked about, I don't think about it much. I'm not saying that's the wrong approach because if the type of, so rote memorization, and this is going into different, this is a massive theory. This is a massive generalization because different countries have different things. But I talk about the UK specifically, you know, rote memorization used to be much more popular 100 years ago than it is nowadays. That was like, that was the philosophy of education. Now, due to just research in the space, but also technology, there is, there is more emphasis now on other types of education thinking, which is actually concept or problem solving, conceptualization. You could argue the AI that it got rid of the memorization angle because it's not needed anymore. Equivalent of like, can, how many people can read maps nowadays? So I can read a map because I had to grow up and read it. And actually, yeah, I can too., and it made me train other types of thinking and spatial recognition. But if I never had to use that, doesn't mean I'm not smart. It just means I'm using it for other types of application. So go, go back to your point. I see a future where, in the best way, you see how the AI should be augmented by learning.

Mm-hmm.

I mean, emphasizing other types of things. But right now we're in a weird situation where you're right, which is, I'll give you an example of coding. Coding before LLMs happened to you, a lot of it would've been Okay, I tried it, it didn't work. I might go to Stack Overflow and try to figure out the answer. If it's not there, what do I do? I need to actually sit down and try and do this thing. That happened, well, happened quite often with many of my students. Now, with AI, I thought it's already there, it's solved for me. The worry people have is that at some point you'll come across problems that AI does not know. And you may not have insufficient training to stretch your intellectual muscles to say, okay, how do I handle uncertainty at a scale that I had to build this from the ground up again? I'm not saying that's right or wrong, I'm just saying it's quite different. So if you're entering the workforce now, it's a different way of education and how we reflect that is going to be quite different, I would say, not good or bad, but I think if I had a meeting, it would be, okay, how can we best augment, use AI to augment my capabilities through training, isn't just rely on this thing, but also maturity by thinking, okay, I, I see it as a tool, or I see it as a, as a companion, I see it as a way to augment my capabilities. So I can actually use my learning process and my learning tools to do this. Right now we're in a clash between old-style education and the rise of technology that kind of authorizes a lot of that training itself, if that makes sense.

Yeah, it does. I think it's interesting. Like, I feel encouraged talking to you about it, obviously. Like, and, you know, part of that I'm sure is because you've got a commercial aspect to it, but also the academic aspect. Like, you've got obviously a deeper knowledge of AI than I do. I mean, let's be clear. But I do feel encouraged by it. And I have some conversations where I'm less encouraged and somewhere I'm more. So I hope that more people in industry are thinking the way you are. More frankly. My biggest fear is not the AI, it's the, it's the economic pressures to make the AI everything before it's ready and before it's fully baked and before people have had all these security questions answered and before there's companies in place to, to do the interrogation of, you know, the LLMs and of the, the data models and all the data that's behind it to make sure that, you know, we're not being poisoned, that we're not being set up to be overly reliant on systems that aren't what we think they are, basically. So listen, I, I, we gotta close up here. I gotta, I gotta, I got a little behind cuz it's been a very interesting discussion and that's fine. Maybe we'll edit some out, maybe we won't. But I do wanna get to the closing question here. You know, the name of the show is Unspoken Security, and with that in mind, tell me something you've never told anyone before, something unspoken. And it means you can't have told AI either. So it's gotta be something that you haven't told anybody, including AI. I can't go look this up in one of the AI models.

Sure. Yeah. And again, if you're using AI, guess what your data is actually in the organization, by the way. That's not So, um, it's, no, I'm pretty transparent with myself. So maybe one thing that I wouldn't typically tell people, it's not relevant, is again, this is very UK-centric. I'm trying to think of the comparison in the US. So I'm a university professor. I'm a scientist. I love being in innovation. I don't have the equivalent of like a full high school diploma in the equivalent of the US, if that makes sense, due just to my background. Wasn't necessarily the best location, let's say, at the very, very least. Okay. I've already gone into this type of learning when I was actually a little bit, I was like 16, 17. I realized I checked a lot of stuff, like environment-wise. Why even in my own students, my researchers, you don't need to be smart to be a scientist. That helps. It's actually a lot of other skills that are super important. Let's talk about any type of job in the space. It's like ideas of being kind, having perseverance and ability to learn. That stuff is by far much more important. Education formally depends on what you do, but about it. So I do think about what would have happened if I did grow up, have that sort of thing. But I think people would be quite shocked by saying you're a professor though in this space. You have a great background in academia. You write all these research papers. I say, yes, but when I grew up, that was not the primary focus, if that makes sense. And again, a lot of that is in the US in terms of where you grew up in certain areas in the country or maybe certain regions. So the violence is not a culture question as much as they should do. Which is why part of my job is actually trying to catch people who especially are first generation to university to say, actually, you do belong here. Or if you're not university, actually, you can actually learn this stuff. You don't need to lay it on your back.

Yeah. I mean, that's one of the better ones I've gotten on this show, to be honest with you. I think I obviously would not have thought to myself, I mean, he's a doctor, he's got a PhD, clearly he's, you know, well educated. I guess the closest equivalent I could think of in the US would be, well, I'm an example, although I don't have academic stature you do. I have a GED. I don't have a, I don't have a high school diploma. I didn't graduate high school. I was thrown out my senior year. I eventually got a GED, which is the equivalency test. Since went on and got, you know, associate's, bachelor's, and master's. But, and I've met a few other people in our industry that have similar paths because of any number of reasons that can, you know, derail your education when you're young or you derail yourself because you just don't care enough like me. But I think that might be the closest thing it sounds like from what you're talking about. And, you know, like you said, it just shows there's different paths to get places. It's about it's about aptitude and attitude and, you know, drive and determination more than, you know, there's no one specific path. Some people have easier, I don't want to say easier, but different, more traditional paths, I'll say.

No, you're wrong. I love education. I actually, I'm always learning stuff now, but it wasn't until I was a little bit older that I actually was able, the environment allowed me to do so at the time. So people who are a little bit younger listening, start doing it now. It's super exciting. But if you are maybe post-high school, generally you can start today and actually really exciting. Like every day I'm excited about learning stuff.

Yeah, no, I can appreciate it. I, I, I'm much more interested in learning now than I was when I was young, frankly. I just wasn't very excited about all of it necessarily. And as you said, I, I dig into stuff and learn all sorts of things. I'm crash coursing myself on AI right now and I'm both very intrigued by it and interested and also losing a lot of time and energy and sleep over it because you just keep learning and it's just, it changes every day. It seems like I'm onto another thing and another thing and I keep learning stuff. I can't keep up. I can't learn it fast enough. And it fascinates me. And you know, that's what draws conversations like this. So listen, I want to thank you for coming on, Peter, and taking the time. Before I let you go, we didn't talk much at all about, you know, your company. I think, I think you deserve a plug in here a little bit. Do you want to talk a little bit about, you know, about MindGuard and what you guys do and where somebody can find you if they want to talk more about that as well?

I can do it very briefly. MindGuard is a company that looks at security for AI. So we do automated red teaming and pen testing AI systems. So if you're someone who has AI or starting to build AI and you want to answer, what are my vulnerabilities? Come talk to us. We have a product that'll help you up your labs.

Brilliant. I appreciate it. So it's mindguard.ai. Please reach out to Peter. You know, thank you very much for taking the time to come on today. It's a very interesting discussion. We'll have to have you come back on. I definitely want to talk about the unlearning and machine unlearning as that comes along. I think that's very very interesting. But again, thanks for taking the time to come on the show today for talking about, you know, can, can people hack AI and all, all these risks, both, you know, from a technical standpoint and societal. I know I threw you a lot of curveballs and I appreciate you, you know, taking all those in stride and, and giving us some really strong answers. So with that, I'm gonna go ahead and wrap up for today. So for everybody listening or watching at home, please, if you like this, you know, make sure you, you give us all the likes and the, the reviews and the subscriptions and share it with other people and let the world know so we get more people they get a chance to hear, you know, stories like Peter's and others. If you don't like the show, then don't tell anybody. You know, ping me though, and maybe I'll try and see what I can do to make it better. And with that, that's going to conclude another episode of Unspoken Security.

That's a wrap for this episode of Unspoken Security. If you enjoyed this episode, follow us wherever you listen to your favorite podcasts. See you next time.