Protecting us with the Database of Evil

What we do is we basically we combine this, like, very, very deep subject matter expertise with our technology. So we're a technology company and yeah, we also have sort of experts in the field in the domain, like experts in the field of researching human trafficking and really understanding that space or in misinformation and different types of misinformation and in hate speech and in terror. And so they speak the languages, they research the space, they understand it, they know the key players, they know the different organizations, the keywords. And you know, this is an adversarial space. It's constantly changing. And so they make sure that they stay up to date. And then what that means is that us on the data side, we can basically take their ideas, take their knowledge, and then engineer features out of those. Right. To really translate the human knowledge into our models. So then we can go out and automate that and do it at scale.

Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive and accessible to everyone. Subscribe now if you haven't already. Head to PracticalAI FM for all the ways. Special thanks to our partners at Fastly for delivering our shows super fast to wherever you listen. Check them out@fastly.com and to our friends at Fly IO, we deploy our app servers close to our users and you can too. Learn more at Fly IO Foreign.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I'm joined as always by my co host Chris Benson, who's a tech strategist with Lockheed Martin. How you doing, Chris?

Doing very well today, Daniel. How's it going?

It's going great. So yesterday was voting day here in the US and I did go to the voting place and it was interesting because in line I could hear people talking about cyber threats to the voting machines and other things like that. And so my mind was already actually thinking about these things because we have a really interesting topic to talk about today that's in that same vein. We're privileged today to have with us Matar Haller, who is VP of Data at Active Fence. Welcome, Matar.

Hi, thanks for having me.

Yeah. And Active Fence, I've read a bit about it and the sort of website talks about this barrage of threats that online platforms are susceptible to now, which Active Fence is addressing in various interesting ways, which we'll get into, but I'm wondering if you could kind of give us a picture. Like, if I am going to run an online platform of some type, maybe it's not even Like I'm likely not going to start and run the next Facebook, but I might very well start and run some type of software company that provides an online platform to do something. What should be on my mind and what's the reality of kind of online threats that I might need to be aware of if I'm getting into that space?

Yeah. So first of all, I think there's one thing to think about is that anytime that you have a platform that has any type of user generated content, whether users are uploading photos or they're chatting or they have comments or anything like that, you're going to have tons of data very, very fast. And just it's prime for people to, you know, post wonderful things, but also some really, really dark things which we've all sort of seen and been exposed to. And so one thing to keep in mind is that trust and safety, just basically safety online, it's not really a nice to have anymore at this point. It's a competitive advantage. It's kind of a basic expectation. Right. So users are expecting it, advertisers are expecting it, parents are expecting it, the public expects it. So if you're going to spin up a platform, so first of all, best of luck. And second of all, you need to keep this in mind, like from the get go before you sort of find yourself down this, this rabbit hole. One thing that I think is really important to keep in mind is that although trust and safety isn't a new industry really, it's really now finally becoming something that people are aware of. Like I said, it's this basic expectation now. It's not only users, but also regulators and legislators. There's new legislation coming in that's making it even more at the forefront. And the basic sort of content moderation that is out today doesn't really make the cut. To follow up, the second part of your question about sort of like what kind of harms are out there. So online harm is really multidimensional. We can see it in different media types. So we've seen it in games and merchandise sites, chats, text, video, audio, things like that, across many, many different languages and also different types of violations. You have white supremacists and terrorists and human trafficking men. These really, you know, painful sorts of things also goes into like misinformation, disinformation, fraud, spam, cyberbullying and so forth. And so it's this really, really complex space that you need to have a deep understanding of, to understand how to address.

And in the sort of, I guess up until this point, you were talking about, like content moderation and how it has evolved over time, but is still kind of lacking in the sort of traditional sense. What does it look like? I mean, is content moderation like people might have in their mind, oh, I have a blog, right? And I'm going to choose whether I allow people to like, post a comment or I have to approve that comment before it's posted, or something like that. So this sort of content moderation, is it in your opinion, kind of as where we sit today, is most content moderation sort of reactive at this point? Or how do you view, like, how are most people approaching the problem right now? Maybe? And why is that lacking?

So there's kind of different levels, I would say, of content moderation at this point. To just sit and moderate. Every single comment gets out of hand really, really fast. And so there is some level of automation that started being introduced. The first sort of basic level is let's go out and look for keywords, right? Like, I don't want any slurs on my platform. I don't want anyone calling anyone any of the like words. You know, I don't want that there. So I'm going to ban all those I know that people might have. And then, and then people kind of get a little trickier, right? And so they'll say, well, you know, what about if we use an emoji? And so there's different kinds of emojis or combinations of them that also can be used in a hateful way. And so you can say, okay, well I'm going to, I'm going to ban those. And I'm going to ban these specific keywords. And I'm going to ban these engrams because, you know, these phrases are bad. Like I hate Jews or I hate what you know, so I'll ban that. And that, that works, okay. But very, very quickly you get to these cases where keywords are either insufficient. So like I said with emojis or with leet speak, so you can write for people that aren't familiar. It's basically taking a word and replacing letters with numbers. So Adolf Hitler can be written pretty much all in numbers. And so it's kind of to evade detections and only on a need to know basis sort of thing. There's also in the keyword space you have numbers. So like 1488 is a white supremacy number, 88 for Heil Hitler. And 14 is like a phrase that they use, like the number of words in the phrase. And so you can say, okay, well, I'll add all these to my dictionary. But then you get to the place where you say, well, what about someone that's telling someone else, don't call me some slur, don't call me that. And so now they're using the slur. Right. Is that something that you want to necessarily ban? Maybe in some cases, but in other cases you're going to need sort of a deeper understanding of how language is being used before you just go out and outright ban it. And so you see this sort of like evolution in terms of the ways or the approaches that platforms are taking to moderate the space and sort of looking at the context in which language is used is sort of the first.

Step in that, I think I just realized. And that was a great explanation, and I just realized how sheltered I am because several of the things you referred to I just didn't know at all. So I guess I'm very sheltered. But I'm curious, when we were first starting the conversation a few minutes ago, you also mentioned misinformation, and we were just kind of diving into kind of some of those specific use cases on hate speech. How does misinformation. Because we've seen, you know, we've been dealing with hate speech for a long time now. But, you know, misinformation, you know, in the last few election cycles has really become a huge issue. And obviously in national security issues and things like that, it's big. How does that fit in? I think in my mind, I've kind of thought about there's hate speech and there's misinformation and all. Is there a connection between them all? Are they bound together in some way, the way you see it, or are these distinct, separate kinds of things? How do you think about it? How do the folks at your company think about it?

That's a really interesting question to take these two examples of these two specific violations. And I think violations are sort of on this spectrum, right? And so you can think about these sort of more evasive violations. So things that are, you know, that kind of are more difficult to find or require subject matter knowledge. So hate speech, right? You need to know these keywords. You need to know these things. And then you also have these just sort of more like common violations where some of them are like, well, maybe it's not even a violation like nudity or profanity or things like that, where it's just more, more out there and everything kind of lies almost sort of on the spectrum. And you can go like spam and fraud and so forth until you get to like the really dark, you know, like, like child safety and abuse and things like that. And with misinformation, it's actually interesting because it's not really trying to evade. Right. Like that's the whole point. On the other hand, it's really tricky to find and to understand. And there's lots of organizations that do really wonderful work of fact checking and of keeping up on the trends and really identifying misinformation. And there's also sort of these techniques that we can use to. Once we've identified sort of like a specific type of misinformation, then we can use it to find in that it's going viral and so forth.

It's a real struggle to understand like kind of how to put that in context. Because you could say things in a misinformation context that are, you know, there's no hate speech in it, you know, explicitly. There are no ban. There's none of that, is there? And yet, as we've seen in recent years, it can do great harm. So it seems like a very hard target to go after and be able to mitigate it in a, in a sane and reasonable way.

Absolutely. I think that's one thing that's really unique about Actefense and what we do is that we're. What we do is we basically, we combine this like very, very deep subject matter expertise with are technologies. So we're a technology company and yeah, we also have sort of experts in the field in the domain, like, you know, experts in the field of researching human trafficking and really understanding that space or in misinformation and different types of misinformation in hate speech and in terror. And so they speak the languages, they, you know, they research the space, they understand it, they know the key players, they know the different organizations, the keywords. And this is an adversarial space. It's constantly changing. And so they make sure that they stay up to date. And then what that means is that us on the data side, we can basically take their ideas, take their knowledge and then engineer features out of those. So really translate the human knowledge into our models so then we can go out and automate that and do it at scale. The reason that it's so interesting is because as you all know, models drift, they decay. And so you can go out and you can retrain your model and get new weights and you're great. Except if you're in an adversarial space, then not only are you drifting, but your reality is like it's so not stationary that it's just changing from underneath you. So as it's changing from underneath you, you need to hurry up and re engineer your features. And so we're constantly engineering new features, retraining our models, and also thinking about just like, what else can we possibly extract from this data that's coming in. Right. We're analyzing text, video, audio, everything, basically anything that we can get our hands on and just really milking whatever we can out of it.

That's super interesting. I have so many questions. Really interesting, really interesting technology. But also like the infrastructure and management part of that is, I'm sure a great challenge. But I'm glad you brought up the modality thing. And I also saw, you know, on your website you talk about different languages as well, which it seems like this is definitely now in terms of the way people communicate online. Like you mentioned emojis. I was also thinking of like GIFs or GIFs, depending on who, who you are, or posting memes with text in the image. There's also, of course you're talking about videos and audio messages. All of that. I guess as a more general question, is language, but sort of multimodal language, Is that your primary area of, of research or are there other things outside of communication in terms of the threats posed to online platforms where someone's not trying to communicate a certain messaging or something, but it's still a threat to the platform in one way or another? I guess maybe spam would be an example of that, but I don't know if you have other examples or is it really from your view, A lot of what you focus on is the communication and language piece of.

So the goal, it's not necessarily language. And in a second I'll talk a lot about contextual AI and what that means, but really our goal is to enable users to be safe online, to have a safe experience. Right. I'm a mom, I have three kids. I started working at defense and I said, oh gosh, my daughter is not getting a cell phone until she's 35. Forget about it. Because you're suddenly exposed to all this. And then you say, but that's why what we do is so important, because that's our whole goal. It's not only about language. And that's just one form of communication. There's lot of things out there and like we're a bunch of concerned parents, like, let's really make this a safe. Like the fact that Chris is, is still sort of in this sheltered bubble is amazing. Like, I want everyone to be in this sheltered bubble. Right. And so that's. That's kind of the idea.

I think I just need to correct it. It depends. There's some bubbles I'm sheltered from, I think, and there's some. I probably am not probably.

So.

Yeah, yeah. Like when you were talking about the hate speech, the specific numbers that meant stuff, I was like, I didn't know that. So anyway, I didn't mean to cut.

In, but why would you? Yeah, so, I mean, so language is only one part of it. And I think one thing that we really get into is context. And so let me take you on a journey through context. And we'll end up at memes, which to me is like, sounds great. So we were talking about the language context, right. And so how keywords and N grams, they just don't cut it. Right. You need language models, so Transformers and so forth to really get an understanding of. Of what is being said in the context in which it's being said. And so those are the kinds of models that we end up training and that we have data for. We have, like I said, we have, like, our subject matter experts and policy experts that are able to ensure that we're capturing things that are sort of on the edge and border, because that's where things get interesting. Right. And that's where I want to be able. That's how I'm able to get the difference between I'm proud of being a whatever, because I'm, like, reclaiming that word versus, you know, you are a. You're not allowed here. And even within the hate speech of language, there's, you know, insulting hate speech, not insulting hate speech. You know, like, hey, wasn't that a great KKK rally yesterday? I really like your proud boy's tattoo. Right. It's hard to catch those things.

That's a surreal statement right there. That example. I just.

That's never been said on your podcast, right?

No, it's not. It's never been said. But just like you said that, I was like, wow, I'm. I'm not talking to the people that are saying things like that. So anyway, sorry, go ahead. I just. It's. It's novel to me to hear some.

Of this perspective, but then we can sort of go to the level of, okay, so let's say I. We're now in the image space, right? And so one thing that we're able to do, and again, because we have this, like, deep subject matter expertise, is we're able to search for logos, right? So logos of terror you know, like falcons, like rare terror groups, small, that are hard to find. We know this space and so we're able to go out and do logo detection, find those particular logos, identify things. And then you say, okay, great, here's the video. Found the ISIS logo, great, check, tear. And you say, well, wait a minute, but there's also the CNN logo here. So suddenly, even though it's a snippet from isis, the context in which it's used doesn't make it violative. It's suddenly it's interesting, it's important, it's historical, it's whatever. You can see the same things with like videos of Nazis marching, right? Like sometimes that's glorified and sometimes that's just like historical. You know, it is what it is. So that's another level of context. We have to sort of look at the context of the, you know, one signal out of the image isn't enough or out of the video. Another thing that we like looking at that, you know, it's important to look at, is that you can look at the context in which the image is being used, right? What is the title? What is the description? What are the comments? We have an example that I like using where you see sort of non violative text, I love him. And you're like, okay, that's fine, who cares? And then when you zoom out, you see it's like, I love him with a picture of Osama bin Laden. And suddenly, like, that's suddenly more interesting, right? Suddenly it becomes violative. So you can't just take any one piece in isolation. Or there's an example of some chef that's like, it's hard to do this on a podcast, but showing knives, and he's like demonstrating these knives and showing these knives and his hands are all cut up because he uses knives. And if you do just object detection, it screams at you like, weapon, weapon, weapon. Like, oh gosh, this is like this terrible video. And then you analyze the title and description and the comments and the channel and everything along with it, and you're like, no, he's teaching about knives. It's a chef video. Really not interesting. And so looking at things just as keywords in a sentence aren't enough. Also just looking at an image by itself isn't going to tell you whether or not something is problematic. And that's this idea of contextual AI that we think about a lot is what is the context in which something is used? And context can mean lots of things. It can also mean the policy so different Platforms have different policies. Some platforms will say, baby's first bath is like child abuse, like you cannot have it, you know, child nudity. And others will say, not a big deal. That's like another level of context that our models need to deal with.

I mean, just knowing about where NLP models or other models fail, like, this area of, like, sarcasm and humor is so difficult. And there's this further distinction that you're drawing out, which is, well, there's some memes that are jokes and sarcasm, right? There's some memes that are jokes and sarcasm to the point of being very, very harmful. And also that's kind of tied into the context of where they're, you know, where they're put or the timing of when they're put somewhere or something like that. So I'm wondering if you could kind of break down, like, as you're stepping into addressing some of this, you already mentioned frequently you're updating new features, there's new behaviors that you're seeing that didn't exist before. So let's say that Active Fence starts to understand that there's some type of new behavior that's harmful or something like that. What is your process and how do you think about going from knowing this is happening to detecting that this is happening in a repeatable sort of way?

So there's a couple of different ways that we're basically staying up to date. So the first is really, really close contact with subject matter experts that are out there gathering information, intelligence, researching, collecting data, building, like, keyword databases, looking for particular bad actors that frequently post things. They're really there. And so we're frequently talking to them and understanding what was it that made this violent? Or they'll send us a report and be like, hey, this is a new hate group, or, hey, this is a new meme, and so forth. That's one thing. The other thing is that even within our models, we're constantly getting feedback. We have something that we call the Database of Evil, which is, I mean.

Might as well call it what it is, right?

That has to be, though, the best name I've ever heard. It's the Database of Evil.

And it's true to its name, I believe it. And so we keep that updated, right? So we have data that's coming in, we score it, we give it a risk score, which is essentially the probability that it's violated for some violation. And then we have trained analysts that review it, review the score, and sort of can say, yes, no, no. Anything that's verified as being Violative, it goes into the database of evil. The database of evil is sort of used for a few things, one of which is new content that comes in. We can say like, well, have we seen this before? Do we know it? Right? Things that we've seen a lot. They're more like, you know, versus things that are. That are brand new. And it's also used because as we take this feedback, we're constantly, like I said, we're retraining, right? We're learning. And that's like, those are the small adjustments that we can make to our models. And so it's this idea of like constantly getting feedback, both just, you know, from researchers that go out and find things from the data that's coming in is being scored. And then we're sort of retraining on top of that. And then, of course, we have our database of evil.

So let me ask a question. Obviously, as you pointed out, your database of evil has a lot of really explicitly evil stuff. But I'm also imagining that there are gray areas that you kind of mentioned the baby's first bath kind of thing. And baby, you know, that would depend kind of on audience, on whether that was. If a family member showed me, well, we have a new baby in our family. My niece has a new baby, and if she showed me a photo of the new baby having his first bath, that would not be offensive to me. But there are contexts for posting it online, it could become offensive and such. So with these types of gray areas and the fact that you can have one set of content that has a bunch of different, I don't know what you would, you know, acceptable rankings, if you will, you know, depending on. On who is viewing it and what the context and all that. How do you approach making sense of all the gray area? Like, you know, when. When you're. When there is everything from perfectly fine to absolutely not fine, and it's all valid for the same thing, right?

That's a really pertinent question. And it's something that we're dealing with a lot. Part of it is sort of. And we haven't. I don't think that we've completely cracked it. But one thing that we do do is that we can also have database of evil where evil is relative for the client, which also leads to this idea of customized models per client based on the feedback that is coming from them. Where basically you can say, okay, and we have this now we have two clients. One is baby's first bath is violative and one it isn't. And so we already are juggling this with different levels of human intervention to get it to really perform, because that's what's training it to get it to that point. And there's even other examples where if we take baby's first bath and things like that, people are like, oh, but it's so clear what child abuse and pedophilia are, and clearly it's not. And even things like asking someone, are your parents home? So if it's a conversation between two children on a chat room, that's totally fine. But it can take a much darker turn when you suddenly see that it's, you know, a user that is also in, you know, adult chat rooms or it's being posted like, you know, at around 8:39, 10:00pm when kids are supposed to be starting to go to bed or to be in bed. I don't know my kids, I go to bed early. And so, you know, and so you can start. So, so even then like, you can say, well, I have language understanding and you know, this is a kids chat room. But suddenly there's like all these other levels that you need to take into account to understand if a phrase really.

Is just nothing, just as kind of setting some. I am living what you just described. I've grown kids, but I also have a daughter who is just getting to the point where we're letting her get online and do some of the stuff and some of it's in supposedly safe environments. But then as the nosy dad who's just worrying about keeping his child safe, there are all sorts of gray areas and stuff. So it's fast. And there are also some moments where I'm having a lot of trouble telling whether it is a safe context or not. It's not very clear. And so I can imagine that that is extremely challenging to solve as a technical problem that can be recreated across a lot of different audiences.

Totally. And I think that there's always going to be to some, like, at least when we're like training or whatever, right. There's always. There has to be a human in the loop and for these gray areas. So we do as much as we can with technology and we bring it there. But even if as a parent, you're looking at it and you're saying, you know, I don't know. And so sometimes we can leverage things that you don't have access to. Right. Like we can look at the history of the user or the other, other chat rooms or other things that are going on in the space or how this, you know, who has been in this chat room before, but sometimes it comes down to you just don't know.

So I have a lot of. Chris always knows. I like to ask a lot of practical questions before I get to those. In terms of some of the things you're doing and how you're doing them, I am wondering for a company that's using some of Active Fence's technology, what does that connection look like? One of the examples I'm thinking in my mind is I'm working on a website for some of our partners where people can contribute enlist cards for tools that they're working on, like software tools and technically they could submit anything in that description of that tool. Now I think we have hopefully vetted people that will be submitting content and actually not everyone has accounts and so it's very fairly restricted. But yeah, I'm wondering in that situation or a much more scaled up situation, what does it is working with the right view to have your software platform and then you send off content to some API and get a threat score or something and then you figure out what to do with that threat score. In terms of how does this actually practically work out for a company in terms of. Because I imagine it's complicated. Every company has their different platform. Right. And also like, oh, the format of a Facebook message going to a webhook is going to be different than a blog post being posted to a content management platform. So in terms of data moving around, how does that work out practically?

Yeah, so I think there's kind of two parts to maybe more parts, but two main parts of the question. The first is like, how would you as a user interact with us? And so we have a ui, a platform where you can really see the content that's coming in. You can define sort of codeless workflows where if something is above a certain risk score threshold, then it's automatically filtered out. If it's below a particular risk score threshold and you don't even look at it and then what is your threshold for human moderation that sort of gets between. That sort of gets around this sort of like precision recall conundrum where you're like, well, I set a threshold and I always have to choose what am I maximizing? And you can say, well, let's set one threshold where you're maximizing your precision and another one where you're comfortable with your recall. And then you look in this sort of band and then you can use that to moderate. We also have an API, you can do synchronous calls for text, so near real time, really, really fast. So for chat, if you want to try pre publish and so forth. And we also have async for text and for images, for video sort of and you know, more like where you can send the full context, right? So you can send your content and you have the body of the media and the title and description, whatever you have. That's for the first part of your question and for the second part I think is maybe like a little bit more interesting kind of because everyone, you know, you can build an API, whatever, but what we've spent a lot of time in it and time on is both like optimizing our API, so making sure that it's, you know, very robust and responsive and so forth and then also modeling our data. And so we have a very rich understanding of the world of platforms like how we can model the world of online media or online platforms or user generated content. Pick your favorite term, we model it. We have a very robust and flexible schema where we're able to model a user and it's related to the post that they put and how many likes they have. And it's not always relevant, you don't always need to use it all. But we have this sort of, we have users and we have contents and we have collections and each of those are modeled a bit differently. And so once the data comes in and we ingest the data and it's modeled like this, then we can go ahead and take it apart and score the different parts of it and then through our API, which is able to handle really high throughput and fast, SLA basically start giving you responses. And we've done a lot of work on our backend optimizing. And so we're batching models on GPUs and doing all sorts and you know, picking that, you know, we have all kinds of like code that we've written that basically optimizes like what machine type you want to run on, basically to make sure that everything is like runs as smoothly and as robustly and reliably as possible to get those responses out.

So atar, one of my questions is just in the back of my mind is like just the practicalities of running the type of platform that you're building and the service that you're running. I could imagine, oh, I have this model that is able, model one is able to detect this harmful type of meme and model two is able to detect this harmful type of video. And then all of a sudden you start proliferating hundreds and thousands of models for little pieces of what you're trying to detect. And then another sort of scenario is like, oh, I'm going to try to standardize everything into more generalized models that handle multiple modes of data or try to synthesize things together. How, just practically, as development and research team, have you started thinking about when is this something maybe we want to combine together in maybe a larger model that's trying to address multitask sort of thing or multiple types of data. And then the other side of that is maybe sometimes it is useful to just spin up hundreds of small models and ensemble them together in some way. Any thoughts on that?

Yeah, so we actually do do that sometimes we have models that are just really lean and we serve them as is. And that's sort of for, like I said, for our near real time responses for when we do contextual stuff. So like I said, we really need to extract information as many different ways as we can. So we're looking for logos and we're listening to the audio and looking for known phrases and keywords and language understanding and what have you. And all these smaller models then we do combine into ensembles. We have a feature store that we can basically take from, combine, train the relevant models and productionize them and then add, we call them indicators, but essentially indicators from which then we can get features and then go to a model which is an ensemble of these. We use both approaches based on the SLA requirements, based on also the explainability that we need. We want to be able to explain why something this particular logo was found because sometimes the moderator may not have the full knowledge that we have. And so a big thing that we deal with is how can we take our intelligence and leverage it to the fullest extent? Right. So one way is really to put it in the models and the other way is really to educate the moderators through explainability of the model so they can really understand why things, sometimes things aren't obvious.

Yeah, and I guess you started getting to my other question, which is how and when do you bring in the subject matter experts into the loop? Because I imagine there's certain cases where you're highly probable that this is some type of harmful situation. And maybe given a restricted set of subject matter experts in an area, maybe they're restricted to only reviewing X amount of content per day or something. So is that a situation that you run into where you have to prioritize what you're reviewing with subject matter experts based on some predictive measure that you have and kind of do that in an Some sort of ranked way or do you handle that in some other way?

Do you mean in terms of what the analysts are reviewing for us for labeling or for reviewing the.

Right.

So I'm assuming that there's a limited number of those people. There's not infinite of those people.

There's not an infinite number of those people. And also we want to be very aware of their well being. We care a lot about the well being of the people that we work with. Like Active Fence invests a lot in that. And so specifically for these analysts, I want to make sure that I prioritize what it is that they need to review. I don't always need, you know, I don't need them to review everything. Right. I usually like what I would go for is that I want to review the gray zone. Right. So we do, we have implemented active learning, which, you know, is basically to prioritize what it is that we want to train on. And so that also prioritizes what it is that we want to. We want to review and to label. Because I'm always going for the gray zone. Right? Like what? Like the things that we're not quite sure of that we don't really know. That's where it goes to the expert, right? It goes back to Chris's question, like how do you know? Sometimes, Sometimes you do know, but it's tough. And those are the things that I want to label because those things that are tough are what are where to get, are what is going to feed in and to give, like, you know, my discriminator, the max, you know, the maximum power that it needs and how much.

Sorry to steal all the questions, Chris. I'm just so fascinated by all this, but no worries. One thing that is always on my mind and maybe I wrestle with sometimes is how much do your data science people or the people that are working with the models directly interact with the subject matter experts and share knowledge across that boundary. How do you balance that? Because that's always something I think I struggle with in projects is ultimately it would be great to kind of bring the subject matter experts in all along the way, in every step of everything, because you learn so much. But the fact of the matter is you've got a limited number of those people, but also you have to ship things. So you can't necessarily have the luxury of always having a discussion before you make a development decision. So how do you balance that? Especially because this is such a complicated environment in terms of the subject matter. How have you found ways to balance that? And any Thoughts that you've. Or takeaways that you have from that experience.

Yeah, so one thing that we did is we actually embedded subject matter experts, like researchers into our dev teams to sort of be part of the process. We also have, you know, our analysts that are labelers. They're, you know, they work really closely. They're just, it's part of the same group, and so they're not out there. However, again, it's a limited number. It's, you know, there's, there's. And it's a limited number of violations. There's, you know, we're constantly being exposed to new stuff that we have to handle. And there, it's just a matter of, you know, relationship building and of doing, you know, check bases and constant feedback. So, hey, this is, you know, like, they're kicking off new projects. We come, we learn, like, what. What are you guys doing? Because a lot of times they're learning on the fly too, right? They have this new, new trend, new thing that they're learning about. And so as they're learning, we're trying to gather as much as we can from them and then just a constant, like, feedback, like, you know, how does this look? How does this look? Is this there? Is this not. But I think, like, the key was embedding them with us. We did have situations where basically we wanted to develop models for things that we didn't want to expose our data scientists to and that only a very, very few number of people in the company can be exposed to because of the nature of the violation. And there, that was much trickier because there was complete dependence. The data scientists. How do you build and train a model without looking at the data?

So I'm kind of curious because. And it changed the question I was about to ask you just a little bit with what you just said. So I'm going to combine two things. The sense that I got, because you keep talking about going to the gray area and stuff, is that almost the core of your research effort is to kind of replace the human intuition that's necessary, you know, early on to identify the nuance that's there with more and better models as you're moving forward into that. And so I would almost like a bell curve of difficulty where that gray area is the hardest. But I am curious, and I'm wondering if that's the case. But I'm also curious when you mention those things. Like, it seems like I'm guessing that the things that you really don't want to expose someone to are almost they're not in the gray area. They're way over into the deeply evil side where it's like you're never going to forget having been exposed to that. If you are, how do those balance out? You have that gray area that you're focusing on that you've mentioned several times and then you have that. Those kind of things are they. When you have something that's so explicitly evil and it will imprint a human's mind in a very negative way, are those different problems that you're solving from a date as a data scientist a little bit in that one? Whereas the gray area, there's so much nuance there. Do you see what I'm getting at? Like, how do you balance the approach to building models to handle one thing that's really obviously bad and you just, you know, you don't want to get anyone to it versus developing intuition or an alternative to intuition in the gray area?

Yeah. So I think, and let me know if this doesn't quite answer your question, but a lot of times it can be the same model, right? So you have a model and it knows how to identify because, you know, it gets the distribution of, you know, the data is distributed in some way along the space, right? And so the things that are very obvious are going to be on like one side of the discriminator boundary, right? Like the really like pick your favorite violation, right? And I'll give you examples of like things that are like very, very, very clearly violative and they're on that line. And then when you're training your model, you don't only want to give it just like the really horrible examples. And then things that are just like, you know, puppies and snowflakes, right? That are very obviously not because those are so the distribution between them, they're like, they're so far away that your discriminator is like your decision boundary is just never going to converge. It's going to, it can flip flop back and forth and you know, you'll never know. And so as we're doing it, we're also trying to find things that are like, you know, on the borderline because that's what's going to help us really make sure that we're able to like find the good decision boundary. Because at the end of the day it's really important, like the basic, like the basics is that we have to be able to catch like the isis and we have to be able to catch beheadings and you know, all these terrible, terrible things. But we also want to be able to catch things that are less obvious within that, still within that space. And so that's kind of where I'm talking about the gray area. Just like, you know, for the grooming model you can like, I don't even want to say, but you can think of phrases that are like very obviously grooming, right. Where like you're sexually harassing a minor and it's there in like plain text. Right. But that same model, if you want it to be any good, it's, it's sure it's helpful that it can find the obvious stuff, but you also want to train it on things that are kind of more on the, like closer to the boundary because that's what'll, that's what'll help you in the long run.

I have a quick follow up that as you were talking it was kind of coming into my mind and it's a very human question for a moment. I want to move you out of the data science bit a bit, little, little bit. And just your organization is in a little bit of a unique position on that. You mentioned that there are certain things you want to keep as many of your, of your folks on your team not exposed to, but that does leave some people exposed to some pretty awful stuff and that does impact people we know. I'm guessing that you're one of the people that has had to see some of those pretty tough things to see. How do you cope with that a little bit and keep. You seem to have, be really super grounded in that. And I know like, you know, but as part of the job you're going to have to cope with some really tough stuff. And I know there has been things that I have seen myself online that I wish I just had not seen. I remember early on in the Al Qaeda period some time back and I watched something that was in the news that happened to be out there and I was like, I wish I had never seen that and I will never forget that as long as I live. I'm just curious, in a human sense, how do you cope with terrible things and keep it in a healthy kind of for yourself in a healthy place, if that makes sense.

Yeah, that does make sense. So I think I'm like a very personally, I'm a very mission driven person. It's very clear to me why it is that we do what we do. And I really, really believe in it. Like I said, like a bunch of us are parents and, or you know, or have nieces and nephews or whatever or just care about kids or you know, care about care, about communities and environments. And I just very, very deeply believe in what we as at Act Defense do. And you can really feel that when you're in the office and when you're working with people, everyone is very, very, is very mission driven. That being said, we also do our absolute best to support and protect everyone that works with us. So whether it's like different wellness support programs, we have a psychologist on staff who specializes in resilience and she's available to everyone and she does group and one on one and really helps people build this sort of resilience in the face of what it is that we do. And for me, what personally works for me is just, just understanding why what we do is so important. And yeah, I've definitely, definitely seen things that will never leave me. Never. And I accept that, you know, because I'm doing my part to make everything just.

You're helping the world in that way. I get that.

Hopefully I try. We all try.

I think even though we've talked about hard things this episode, I'm super encouraged and want to thank you and the team at Active Fence for what you're doing. Yeah, it's something that's desperately needed and thank you so much for digging into these problems and doing it with such technical excellence as well and deep insight. As we close out here and we look to the future, what on the positive side sort of excites you about where this technology is headed as you look to the future?

Yeah. So to me, first of all, super exciting that this is. No, like I said, like we started off, it's no longer a nice to have, it's a basic expectation. So I think first of all, to me that's really exciting because people are not taking safety for granted. They understand how critical it is and it's coming from the users. It's not just sort of like, oh well, it is what it is. This is the price I pay for being online. No, that shouldn't be the price that you pay. So to me that's exciting. And on the tech side, I think what's cool is that we're seeing a lot of open sourcing of different, of different models. Whether it's data generation or, you know, audio transcription or Z shots or like all these things that are just. It's like a candy store. Right. Like you can start thinking about all these things that are, that are used for, you know, technologies used for completely different things. And you can say, well, like how can I take these ideas and use them to just extract more signal and to look at. Look at these things from different angles. And, you know, it's an adversarial space, and so it keeps it interesting, at least.

Well, Matar, that it's very inspirational, the work you're doing. I know that there's. It's tough work, but thank you very much and to your teammates for doing the kind of work that you're doing. It was great having you on the show. Looking forward to having you back sometime as you guys surge forward and have some more stuff that you want to. That you want to share with us. So thank you very much for your time today.

Thank you so much for having me. Thank you for caring and for asking really, really interesting questions. I appreciate it.

All right, that is our show for this week. If you dig it, don't forget to subscribe. Head to PracticalAI FM for all the ways and if Practical AI has benefited your life, pay it forward by sharing the show with a friend or colleague. Word of mouth is the number one way people find shows like ours. Thanks again to Fastly for fronting our static assets to Fly IO for backing our dynamic requests to Breakmaster Cylinder for the Beats, and to you for listening. We appreciate you. That's all for now. We'll talk to you again on the next one.