Attack of the C̶l̶o̶n̶e̶s̶ Text!

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog Linode makes Cloud computing simple, affordable and accessible. Whether you're working on a personal project or managing your enterprise's infrastructure, Linode has the pricing, support and scale you need to take your ideas to the next level. We trust Lino because they keep it fast and they keep it simp simple. Check them out@linode.com changelog.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community and follow us on Twitter. We're racticalaifn Foreign.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International, and I'm joined, as always, by my co host, Chris Benson, who is a principal AI strategist at Lockheed Martin. How are you doing, Chris?

I am doing okay. How's it going today, Daniel? What are you up to?

It's been a long week. Helped my. My wife's business move, so I'm a little bit physically tired. But it's been a week. Gave a virtual talk this morning and that was fun, actually. Really good. You know, question and answer and back and forth. So I don't know. You never know with, like, virtual talks. Sometimes they're, like, super awkward, sometimes they're all right. So this one was really good and interesting. What about you?

Same kind of thing. I'm just migrating into virtual talks after just taking a few months off. And yeah, it's interesting as I've been watching different organizations, their different levels of, like, adaptability in terms of their readiness to do that. So I'll always, always fun, for sure. Very supportive. And hey, we had a milestone in our family this week.

Really?

Yeah. My. My wife became an American citizen. She's British.

Oh, congratulations.

She became an American citizen.

Yeah. And pass on my congratulations to her.

Absolutely. It was a big deal for us.

Yeah. That's so awesome. That's a long process, man.

It is a long process. And she knows more American history than any of us, probably more than all of us put together. Now they have to. They test all that, all the things that we learned in third grade and promptly forgot.

So. Oh, I Forgot, right?

She's a pro at this point.

That's awesome, man. Congratulations. That's. That's so exciting.

Thank you very much.

Yeah. Well, this week I'm pretty excited. There's sort of a gap in our discussions thus far on the podcast. We've talked about a lot of things, but there's still things we have not talked about, apparently. And every once in a while, like, I see something talk about, like, oh, we haven't talked about that yet. There's just so much to cover. So today we're going to be talking about adversarial examples and attacks, and we're privileged to have with us Jack Morris, who is a researcher at the University of Virginia and an incoming AI resident at Google. Welcome, Jack. Hi.

It's great to be here.

Yeah, thank you for joining us. It's really, really great to have you here. You've done a lot of stuff already and you're working a lot in open source and there's a lot to discuss. But before we jump into all that, could you just give us a little bit of, you know, your background, how you got interested in AI and, you know, got eventually connected with Google?

Yeah, sure. So I recently graduated from the University of Virginia and I studied computer science and math there. And while I was there, I did this internship at Google. Actually, it's kind of a funny story. Have you ever heard of Google fubar, that program? No, maybe I have not.

I don't recall it. Tell us about it. Fubar us.

Okay, so I'll tell you the story. So basically I was in my sophomore year at college, at uva, we call it second year because, like that. So I was in my second year. I have this personal website I think I was working on. I was searching up something related to a software project I was working on on Google. So, like, I think what I typed in was something like Python List comprehensions, something related to that. And I typed it in and then it's hard to explain in audio, but basically, like there's the search result page for Google, like all the results, and it kind of like animated, like it shrunk up and slid over and then kind of like behind the search.

Oh, I've heard of this.

There's a terminal.

Yeah, yeah, it's. It's. Yeah, it's like an Easter egg. Recruiting.

Yeah, exactly. Yeah, it's pretty funny. It's not like they don't send a lot of recruiters to my school or anything. So it was very, I guess, like, serendipitous for me that I got that pop up. So it's like, it's a terminal window behind the Google search page. It's like just like an Easter egg in a video game. And it says, we think you're speaking our language, want to take a test? And then I click yes and you make an account. There's this little website that's just a console in the browser and it's coding challenges. Like now they have the code and stuff like that. It's that kind of thing. And there's different levels, it gets progressively harder. I remember doing it for a few weeks and a few of the problems were they're not like interview coding problems where you kind of think for a while then you get it and you can write it out. They took me like several days I think just to sort of figure it out and try and optimize and stuff. And I got to like level four and then I just couldn't figure one out. I was so excited to do when I got it and I was kind of bummed but I just could, I just couldn't figure this problem out. And they all have these really long write ups. It was something to do with rabbits and there are all these allocations of rabbit holes and they were storing things and different. It was like a hard problem and a really confusing write up. And then I just kind of gave up. And then I guess it turned out you didn't have to, you know, solve all the problems to get an interview. And I got an interview and I did an internship. So that's how I originally got connected with Google. And then more recently I applied to this program called the AI residency and did some interviews for that. And my eventual goal is to get a PhD hopefully and something related to computer science, artificial intelligence, linguistics, not that sure yet. But next year I'm doing this one to 1.5 year fellowship thing at Google which will be really awesome. It's just a research internship basically. And as to the question of how I got interested in AI, originally I was really interested in just some things on the application side. They're all pretty pertinent to nlp, but I didn't really know at the time. I think my original idea was I worked on a few different open source projects and I wanted to make a website that aggregates like medical literature. Like it seemed like there's all this research that's just open online on Google Scholar, but it's pretty hard for the average person to sift through and aggregate and summarize. There's all these sort of data Processing, what do you call it, Information retrieval tasks. And so I thought I was wrong about, I guess, like, how far along we are in solving some different NLP stuff. Like I thought that would be sort of like a plug and play type of deal, at least on a small scale for certain types of research and things. And I tried it for a little bit and I remember I tried this thing two and a half years ago, was when I first started getting interested in AI and I tried to use these skip thought vectors to encode sentences.

And what is a skip thought vector? Just for those of us who don't know.

Yeah, it's one way. It's now, I guess it's no longer kind of like a state of the art method, but it's a way to take a sentence and encoded into a vector so that you can compare that vector to other sentences to see if they're similar or dissimilar.

Gotcha.

And then I kind of was slowly realizing basically, like, wow, these things. Not that it didn't work, but that, I don't know, there's just still a lot of problems to be solved. And I got interested in some more specific stuff. And then most recently I've been working on this idea of like robustness or. Yeah, like this security side, almost like kind of like trying to find flaws and NLP models the same way that people are interested in finding them for computer vision models specifically.

Yeah.

Which is the buzzword is like adversarial examples. And it's kind of a lot of people debate whether or not that's really applicable terminology to nlp. We can get into that later. But anyways, that's the kind of field that I've been interested in recently.

Yeah, I remember. And I think I've mentioned actually the same episode of a different podcast on this podcast before, but the NLP Highlights podcast, which I listen to, they had an episode about behavioral testing of NLP models with Marco Ribeiro. And he was talking about these tests that he did which were really like right along the lines of what you're talking about. Like, you know, if you, if you have a sentiment analysis example that's like, I love the United States, it's great. Or something.

That's a timely example right there. Given my family.

Yeah, yeah. And you put that into like the commercial, like offerings for sentiment analysis, like products that are being sold. Right. And so you do that and then you just switch out the, like, United States and put in like turkey. I love turkey. And it's great. Right. It'll actually like Break the system and you won't get positive sentiment anymore. Just because like so much in the training is like biased against like turkey for whatever reasons. So yeah, I don't know. Is that kind of what you're talking about with adversary? Maybe we should just back up a second and think about like adversarial examples in general. What, what do people generally mean when they're talking about adversarial examples?

So the first people that were researching adversarial examples were all based around convolutional neural networks and images specifically. And you guys have probably seen the examples where you can take an input image that's classified correctly, say for some imagenet classifier classifying different animals. There's a picture of a panda and it says, okay, this is a panda, I'm 99.8% confident. And then basically they uncovered these sort of like blind spots in the really high dimensional input space that point to other classes. And there's a lot of different research into why that exists. But if you add a small, tiny little image onto the panda image so it's so small you can't even notice it with your eye, it's maybe a pixel or 2 pixels out of 255. You add it to the panda, you end up with an image that to a human can't be distinguished from the first image. But if you add noise in the correct direction, it's totally misclassified into some other class. So I think the quintessential example is like the panda. And then you add this tiny little delta, a small image and then it becomes a given with you know, 99.8% misclassification confidence, which is crazy. And it's a big problem. And it's become, I think that's one of the hotter areas of research and ML I'd say there's all these people trying to figure out why that's happening, trying to propose different defenses for it. There's some pretty cool attacks and defenses. I think in the real world. Like I saw people that are trying to of a facial recognition detectors wear these special shirts. Have you all seen those? Or sweatshirts?

Yeah.

That are patterned in a certain way that convince the camera that they're not a person or they just totally distorts their facial recognition software. It's pretty cool. And so that's a really hot area of research like adversarial examples with convolutional neural networks or other image processing deep learning models. And then naturally one would wonder like whether that exists for text and if so what that would be. And it turns out it's not as cut and dry. It's like kind of a hotly debated and somewhat murky concept. But one hypothesis like you talked about is you can substitute words that maybe don't change the meaning with respect to the task, like within sentiment analysis, maybe if you substitute a noun for another proper noun, you could say that a proper noun for another proper noun, like substituting the United States for Turkey, should never really change the sentiment of any sentence. Right. And I think that's what he was talking about on that podcast and kind.

Of just tying in, you know, as you're talking about, you know, as you were tying into this direction, as you're talking about adversarial in the CNN context, the convolutional neural network context, what would be the motivation for doing it in the NLP context? Like, what are some of the things you could think of that people would be bothering to do that with? I'm just curious what, in your views, might be some of the drivers there.

The first paper I wrote, which is on arXiv, it's called reevaluating Adversarial Examples in Natural Language, was trying to give sort of like a theoretical framework for different types of adversarial examples in nlp. The basic idea was like, okay, we don't have to agree on one set, but we can lay out some sets of constraints that you might agree on. Like, if you substitute a word for its synonym, it shouldn't change the prediction. That would be an example of one partial definition of an adversarial example.

And maybe this is having to do with people in the convolutional neural network space. I think talk a lot about safety in terms of why adversarial examples are worth exploring. From your perspective, since you've done this work in nlp, what makes exploring adversary examples for NLP interesting from your perspective? Is there a safety side of it, or is it more like robustness? You were talking about robustness as well.

Yeah, those are definitely two different things. Like, if you're a company that's maybe putting some NLP model up on the Internet, releasing it into production, on one hand, you want it to work in all sorts of cases, and you don't want there to be some obvious gaping holes in its actual predictions, which would be sort of like the robustness angle. And on the other hand, you want it to be safe. Right. You don't want people to be able to, like, manipulate it in some way. So the reason I brought up the paper was because it includes some specific examples like that. Because I think there's. There's a decent amount of research, but not a lot of discussion about, like, why robustness is important in nlp or like you were asking what maybe the goals of an adversary would be in that situation. So I think one really easy one is there's these toxic comment classifiers that I think are actually in use right now. Definitely on Facebook, they have this whole system for deciding whether a comment is just totally quote unquote, like toxic and needs to be flagged or discarded or hidden or, I don't know, you have to say you're 18 to read it or whatever. And so if you're someone who for whatever reason thinks it would be a good idea to write a toxic comment and then avoid that sort of flagging system, that would be a pretty good example of when you want to run an attack on an NLP model. And you could actually use text attack for that, though you should not. Foreign.

What up, nerds? Jared Santo here, your humble producer. I'd like to tell you about something new. We're beta testing around Practical AI. It's a membership program which we think could be really valuable for the whole community. We call it Change Dog plus plus and it's the best way to directly support Practical AI and all the podcasts, videos and other stuff we create here at Changelog. We have big plans and ambitions for this, but we are experimenting for now to make sure there's interest. That means when you sign up today, you get Practical AI and whatever Changelog shows you listen to now, accept no ads. I guess that means this part you're listening to right now, it'll be gone. We also have some extended episodes planned, bonus content, merch store discounts, and a lot of ideas. But since it's such early days, we are offering memberships at a 40% discount for early adopters. That disappears at the end of August. So head to changelog.com to join today, lock in that discount, get closer to the medal and make the ads disappear. Once again, that's changelog.com we'd love to have you supporting us as a member.

So let's say that our model fails on an adversarial example. You know, what could we do to fix it by way?

That's a great question too. That's a pretty open area of research. I wouldn't say there's a great answer I can give you. Right on. But the sort of naive approach is generate a bunch more adversarial examples and then retrain your model either just on the adversarial examples or on the original training set and the adversarial examples concatenated together. And there's some research that's shown that gives you some improvements. But you can imagine like it's very, it's very different than the case of images. Like for example, if your goal is to replace proper nouns with other proper nouns, right? Like replace every instance of the United States with Turkey, or more commonly, maybe try all the countries and pick the one that meets your goal the best.

I think that you're getting into some interesting areas. One of the things that you started talking about was like generating adversarial examples for NLP models. So it's one thing to realize, I guess, realize that your model fails for a particular type of attack or something like that. It may be another thing like Chris was saying to like, say, okay, well now what? How do I fix that? Yeah, and you're saying you could generate adversarial examples. I know that some of your work and your open source work is geared towards generating those examples, but in general, how do people come up with adversarial examples? What is the range of things that people try to do to come up with these examples? Maybe specifically in nlp, is it mostly hand curated data or is it like, what other things are out there to try?

Yeah, again, it kind of depends on your definition of an adversarial example. Maybe we can talk about that briefly first.

Sure, yeah, that'd be great.

I think that kind of lends a hand to why adversarial examples in NLP are not as well defined as in vision. Because if you have two images next to each other, it's very clear whether or not they're similar and whether the change from image one to image two could be classified as imperceptible. But if you have two sequences of text, there's no imperceptible change unless they're exactly the same. And so if you make a change, any sort of change that you might define as imperceptible becomes at least like a point of argument. So I'll tell you two really popular definitions. One would be with respect to, like, semantics. So if you have sentence one and then you replace some words with synonyms or like you said, with a proper noun, that should generally not change the semantics of the original input. And so you could say if you have two sentences, like if I have one that said, I love the movie Parasite, the best movie I've ever seen, and Then I replaced a couple words. I said I liked the movie Parasite. Greatest movie I've ever seen. You could. A lot of people would say that's invariant with respect to semantics, like they contain the same meaning. So if they have different predictions, that would be classified as an adversarial example. And then the other thing that I was going to bring up, like, another definition is with respect to, like, character level changes. So if instead I just like imagine a typo on the computer, basically, if instead of saying I love Parasite, I spelled love L V, O E D or something like that, just switched around the characters a little bit. It's actually really shocking how many state of the art NLP models will just totally mispredict that for whatever reason. And so those are sort of two competing, not necessarily totally meshed ideas of adversarial examples. One being with respect to like character level changes, or if you just insert a character like L O, V, E, D, Q or whatever. Especially models that are based on this idea of like, entire words have a lot of trouble dealing with those types of changes. So either one of those would probably be classified as an adversarial example by most people. Yeah.

I'm kind of trying to think through, like my own workflow right now. And so like, in terms of integrating or making my models more robust, I guess let's say I'm creating an NLP model. It's probably unlikely, or it seems unlikely for me to sort of plug all the holes in terms of like, things that my model might do that are unexpected. Right. But you're saying that maybe there's some sort of obvious things that we can protect against or make more robust, like with the typo sort of perturbation or maybe not changing the semantics or some type of perturbations like this that we can test against. But then it's probably, I don't know, what are your suggestions in terms of you can plug some holes, but then eventually your model might behave unexpected in a new way that is totally unexpected because it's unexpected. It almost seems like unit testing software or something to me, where it's like you test for the things that you definitely anticipate, but then at some point something weird happens and you have to add a test case. Is that kind of how you would view it or how would you view the workflow of thinking about adversarial examples in developing a model as opposed to not thinking about them? I guess is where. I guess is where I'm at.

Yeah. Yeah, that's a really pertinent question, I think, to anyone who's an engineer actually trying to build NLP systems for real people. And it's not one that's totally solved yet. But another idea that I've heard, I think Ian Goodfellow has gone to talks and advocated for this, is the idea of if you're building a model, adding some kind of output that can identify whether an input is malformed or maybe doesn't fit with the distribution of the training data at all. So that in the case I indicated before, might indicate, you know, a misspelling or some kind of really unnatural misspelling that a user would never produce. Or maybe just a synonym substitution, like maybe using a word that might have the same meaning but would almost never be used in that context by a real person. Which is something I think I see a lot with these adversarial example papers. They might say, okay, if two words are synonyms in the thesaurus, they can always be substituted for each other. A lot of the time that might not be a very natural substitution or something that a human would probably never actually do. And so if you can train a model that has some way to indicate whether an input is sort of like, acceptable or not, it can, I think, alleviate a lot of those concerns, though they're still there. I think in the general case, it would make your system a lot better.

Yeah. So, I mean, you know, I know you and Daniel do a lot of work in this area, and so I'm kind of approaching it as the one who's not actively doing models in nlp, if I'm understanding you correctly. It sounds like there's really a different set of use cases. If you were to compare this to, like, adversarial and CNNs, whereas that behavior is in some sense often maybe nefarious, trying to change a classification for some purpose. Whereas this. It sounds like you mentioned robustness earlier, and it sounds like there may be a lot of use cases where you're helping a user not make mistakes, where you're trying to prevent unexpected behavior, intentional or not, you know, in terms of, like, what the user was trying to do. Do you think that's fair? Do you think that the types of use cases that this might be applied to are fairly different?

It sounds like, yeah, absolutely. And I really like the comparison that you made of comparing, like, finding adversarial examples for NLP models to some sort of unit testing in the software engineering spectrum. That's what the paper that I think he's talking about, the checklist paper, that's what they talk about is applying ideas from software engineering to verifying the usefulness of NLP models essentially. And I think that's a lot closer to what adversarial examples are in NLP and how they're useful to maybe how they are in CNNs.

So I could imagine not far down the road as some of this work is realized, that maybe for programmers, their IDEs or text editors, you know, kind of get some modules that help them from in terms of essentially unit testing what they're doing in a smart way or even for just someone who's not technical, a word processor might have this capability built in so that you know, when you're writing you're making fewer mistakes in that way. So it sounds like there's a lot of help the human possibilities here.

Yeah, it would probably be more I guess on the data side or the training data side. So like if you see a unexpected behavior in your autocomplete, in your text editor or whatever, like why was that unexpected? Did you like give the model like some type of weird like malformed input that like caused it to just like totally fly off the rails? Yeah. And so then it's like, okay, well why did that misbehave? So now we've got to add some things into our training data. So I know I definitely want to get to this open source project text attack, which I actually came across in my, one of my newsletters. I forget it was like one of the NLP newsletters mentioned, mentioned it. So I forget which one it was off to look up while we're talking. But yeah, it seems really cool. So was this open source project, maybe you could talk a little bit about how that came about. Was it because you were trying to like you were in this workflow and you were like trying to add adversarial examples and it was like there just wasn't tooling around it or how did that come about?

Yeah, you kind of nailed it. So I was working on things related to trying to find adversarial examples in nlp. And like I said, there's a lot of kind of like disagreement on what counts. And there's been a decent amount of research into this idea, but it's not very homogenous. It's actually pretty disorganized I'd say. And a lot of people have really similar ideas, but they change one or two things, but they reuse a lot of ideas and all their code and projects are generally implemented in different places. So it's kind of a headache trying to re implement results and compare things. But it turned out that like a lot of the people that suggested NLP attacks were using a lot of the exact same components. So one example is like the thesaurus I was talking about before. So have you all heard of glove word vectors?

Yeah, from Stanford, right?

Yeah, from Stanford, a few years ago. Those are still pretty much top of the line word vectors you can download. So basically they're more commonly called word embeddings. You can download this big matrix that's assigned to, I don't know, 100,000 or a million words from English, and each one has their own word embedding and it's a vector of dimensionality, maybe 300. And they're supposed to encapsulate a lot of information about English. So if you're training an NLP model, the initial layer is probably going to use an embedding similar to that. And glove is kind of like the just very accepted word vector a lot of people used, at least before people started using transformers and subwords. But there's this paper that made these amendments to the glove vectors so that they more directly encapsulate information from a thesaurus. So it's called, they're called counterfeited vectors, not spelled like you would think, like counterfeited. And they basically, if you compare two synonyms based on their counterfeited word embeddings, they should be very similar in terms of their angle, like cosine similarity and antonyms should be very dissimilar, like they should have angles close to 180 degrees. So it's basically just glove vectors plus this pre processing step that uses a big list of synonyms and a big list of antonyms to try and make those vectors closer and further apart. And I bring this up because a lot of people that have created systems to develop adversarial examples in NLP use this base layer of counterfeited vectors. So if you take the example I talked about before, like I loved the movie Parasite and you could look at the word vector, the counterfeited word embedding for loved, and then look at its nearest neighbors in the counterfeited embedding space and you might see a bunch of words that fit in the same context and are actually synonyms. So loved, liked, I don't know, enjoyed. And then once you start to deviate in terms of that angular similarity, then they get less and less similar and there's some debate as to, you know, how similar do they have to be to be synonyms like 0.9 or 0.95, you know, which is a whole nother thing. But all these attacks use those counterfeited word embeddings, like maybe over 10 papers use counterfeited word embeddings and maybe some other components that are exactly the same to generate adversarial examples. And even stepping back from that, the entire process for generating adversarial examples in NLP is very, very similar. So by the process, I mean the process of taking a text input, so a sequence of words, and then producing some other sequence of words that generally, like, fools a model. So taking maybe the sentiment classification example, you have an input that's classified as positive. And the process of finding an adversarial example would be which words can I substitute that will change this classification to negative? And so that turns into a combinatorial search problem. And most people do it in the exact same way and often use the same word vectors and then, I don't know, change a few different things and then release their attack. So our idea was if we break that process down into components, then we can construct the attacks from different papers based on these components.

So you've definitely captured me in terms of interest in adversarial attacks. And so let's say that I'm out there and I've just, I'm listening to this, or I've just come across Text Attack. Can you tell me kind of just describe the library, let me know what I should know about it as a beginner coming into it that wants to use it and, you know, kind of what are the goals that I should keep in mind that the project tackles and, you know, are there any things, you know, that I should not address as well with this library that I'd look elsewhere? So can you kind of give me that beginner perspective?

Yeah, absolutely. So it might help for me to talk real quickly about that kind of, like, system I was talking about, like, the components. Then I can explain the most common use cases, because obviously you can pull out any one of the components and use them for your own purposes. One thing that we really focused on in TextAttack is trying to make it work out of the box. For example, those counterfeited word embeddings. Instead of going to this website, downloading it, unzipping it, moving it, finding out how to load all the data, you just import Text attack and do textattack. The class and just initialize it and it'll download everything for you, which I think is. That is really cool. If you guys know about Hugging Face Transformers, a lot of the TextAttack stuff is built around transformers and tokenizers. And now this data set loading library called nlp, which I'm very grateful for. And we kind of tried to follow the same model. So instead of having all these files, you manipulate yourself, you pretty much just reuse other people's, you know, and it saves a lot of time. So the easiest or probably most common way that I would imagine people use text attack down the line is for things like that, for the embeddings or another very common thing is sentence encodings, which is something I mentioned at the beginning of this talk. Like, there's so many different methods for taking a sentence and encoding it into a fixed length vector. Whether they're very effective or not is a question, but they're useful in a lot of situations. And so one thing TextAttack has done is just sort of abstracted them into classes that work by themselves. And so you could just, for example, if you were doing some project, I don't know, you wanted to look at a bunch of Airbnb reviews and cluster them based on which ones were similar. You could just import text attack and then just call like this sentence encoder dot encode and then give it the list and it would just do it for you, which I think is pretty valuable. So I'll tell you what the components are very quickly. There's. There's four, and we have our own names for them, which I think increases the learning curve a little bit. But there's some benefits, I think, to having our own terminology. So it's all based around this idea of the NLP attack as a system, which is taking the text input, looking for changes you can make to it, making sure those changes are acceptable. And then whenever you have decided you fool the model, you stop. So the first component would be like the. What we call the transformation, which is taking an input and changing some of the words or characters. Like one transformation would be substituting words with their counterfeited word embedding neighbors. And then once you do that transformation step, there's also this idea of a constraint, which is trying to make sure you didn't make any mistakes. So, like, a common constraint is use the sentence encoder. A popular one is called the Universal Sentence Encoder, which is by some folks at Google. And you encode the original input and now your potential adversarial example, and make sure that the sentence encoder also says they're very similar. It's basically like a sanity check to make sure you didn't change the meaning or whatever, change too many characters, if that's what you decide. And then there's two other components. So we have the transformation and the constraints. And you have to define your notion of whether you fooled the model or not. A common thing would just be change the classification output or change the classification output to a specific class. Those would both be examples of what we call the goal function. I think a really cool one that I want to explore more in the future is with sequence to sequence models, like a machine translation model. Your goal might be to take the original output translation and change as many characters as possible. So say you're translating a sentence into French, you would have your original translation. And if you could substitute a word from the input with a synonym and then it produced a translation that was totally different, even just in terms of characters or its bleu score, that would be pretty telling and like probably very bad for your translation system. So that would be another goal function would be trying to minimize the bleu score. And then the last component is called the search method. And that's basically like if you have the input and you have all these transformations, how do you decide which one to keep? Which is important because if you just tried all the combinations, I mean, if you have an input of 10 words and each word has 50 neighbors, you end up with 50 times 50 times 50 possible substitutions that you might want to combine so that the space grows exponentially very quickly. So you have to come up with some sort of like greedy or approximate heuristics for doing that. And that's what we call the search method. So you can combine those four things into an attack, like an nlp, what we call an attack, which is just a search for adversarial examples that meet the constraints, fool the model as defined by the goal function. But there's some really cool other things that come off of that. A big one that I've been talking to people about recently is data augmentation, which is also a very kind of, I would say under research field and nlp. It's another thing that is pretty commonplace in vision. It's almost like everyone does it, you know, if you want to train a state of the art vision model on CIFAR 10 or ImageNet or some other data set, you're gonna do some sort of augmentation to change and increase the size of your data set. So with textattack, if you have this transformation which can find maybe like semantics preserving changes to your input, and you could add on constraints which make sure that they preserve semantics. Then you can end up with like, some pretty good tools for data augmentation just from those two things. And since we're trying to implement more components that would hopefully grow the list of potential augmentation modules as well. And so, yeah, that's something I'm really excited about, is data augmentation.

Yeah, that's really awesome. I know that even in speech it's fairly common to, you know, like, mask out, you know, parts of a spectrogram or like speed up or slow down the audio or something. And that's like, fairly common component. But in nlp, it's pretty much most of the time we're. It seems like people are talking about, oh, we have this parallel data, we have this monolingual data, and that's like, basically it, like, that's all you have. And like, you could augment with monolingual data. But, you know, this is like another. It's another route to that, which is pretty cool in terms of like the open source. I mean, there's been interest in the library. And what is your vision for it going forward? Do you see, like, others you mentioned, like hugging face transformation formers and kind of modeling certain things after that. And I know one of the things that they talk about a lot is, you know, people contributing models to their. Their model hub.

Right.

And actually it's good timing because we had Sasha Russ from Hugging Face on his episode was released this week. So that's a great, great model that they're going. So I, I could see similar things happening here where it's like, you know, there's a new type of, of perturbation or goal function or something like you were saying that people are exploring. Do you envision people kind of like having a way to contribute those models in or like, what is your thought process around how you see the future of the library evolving?

Yeah, that's a great question and something I'm still, you know, we're still kind of discussing and trying to figure out because we've written a lot of code and there's a lot of different features that still could be connected. And obviously once you, once you work on a library for a while, there's just so many improvements you want to make to it. You know, I have this giant list and take my whole life, but I think I could probably break the potential users down into three groups, at least three main groups. So the first one would be kind of what I mentioned before, what I imagine would eventually be just the most common general use case would be using the components because they're easy to use. But the second one would probably be, like you said, people who want to work on researching the robustness of NLP models. So maybe coming up with new goal functions or trying a new transformation. Like for example, I don't know if you created a model that could paraphrase inputs. It's a totally open problem in nlp. But even if you could do it a little bit well, I think that would be really interesting, like taking that input sentence and just paraphrasing it or paraphrasing its individual sentences and trying those different pieces and seeing which one fools the model. That would be someone, and that would be a really cool thing for someone to research. And that would be just a new transformation basically that just takes an input and paraphrases it. And then the third use case would probably be people who actually want to test and potentially improve their models using Text Attack, which is what we added this summer, is what I mentioned before, adversarial training based on just doing an attack and then increasing the size of the training set and continuing training. So hopefully people would be able to like import Text Attack and then test out the robustness of their maybe summarization model or translation model or whatever and get some insights into maybe how it's failing in the common cases and hopefully how they can improve. Improve it.

Yeah, got a note that as you're talking there, I'm looking through the GitHub repo and you really put it together. Well, for someone like me who is new to this, you know, you cover the kind of what it is, you note the Slack channel help with setup, then you have a good section on usage and running attacks there. And then the thing I love the most is I love your section at the bottom on attack recipes.

Oh yeah?

Yeah. I mean, and I was flipping through the links here, you're listing these, these different recipes for attack modes and you link off to the paper or website where it shows what it is and how to do it in detail. And then with the section above it on usage, it makes it really easy to just go ahead and give it a shot. So after we're done with our conversation, I have a feeling I'm going to geek out on this for a little while. So thank you very much for that. Do you envision more contributions being made to this ATT and CK recipe section?

Yeah, I sure hope so, man. We've. There's still papers that I think we could implement with pretty low effort because they have a lot of overlap with the Components we've already implemented. Well, talking about vision, there are a few very common libraries in vision. One is called cleverhans. No other names are coming to mind, but there's. I know there's one by IBM, a one called fullbox. And those are pretty standard tools for researchers that are interested in the robustness of computer vision models. So not even that. I'm super convinced that TextAttack will be that tool. But I'm hoping that just putting everything in one place would make it a lot easier for people to actually do that research and make fair comparisons and, you know, advance the field. So hopefully getting people actually excited about it by making these things easy to use will then lend a hand to people actually contributing. That's the goal. We've been working on this since almost a year ago, about a year ago, but it's only been open source since May, so hopefully that'll come with time. But yeah, right now I think one of the recipes was implemented by the authors and then the rest we kind of did by hand. So.

Yeah, no, I think that this is so well set up and a lot of thought has been put into it. I'm like thinking back to the code that I wrote it in academic research and the reason why it brought me back there, because you have a little logo of a, or a little octopus emoji on text attack. And the code that eventually I implemented my method in and my PhD was called Octopus. But I have no, like, I don't know that anyone be to be able to run my module of octopus just because like, you know, it was nothing near well documented like this or, or anything like that. So I hope that you do get contributions as we kind of end up here. I guess I'm curious just as kind of a last question. You've done this research, but you're also kind of launching into a new thing with your AI residency. I'm just kind of curious, you know, from your perspective, you know, jumping into the field at this point and this new position, what are you excited about in terms of the future of AI and what you want to be involved with? What gets you excited?

It's a great question, Daniel. I think the thing that motivates me the most is the potential for creating systems that actually have some basic knowledge of anything. You know, like there's a subfield of NLP that calls itself Natural Language Understanding, like nlu. And that to me is the most, I don't know, applicable at least. Seems so like philosophically it seems applicable to like my everyday life having systems that can actually do some kind of basic reasoning. The problem with models like GPT3 or BERT, that we, we don't have to debate this right now, is that they have no common sense understanding of anything. You know, at least that's my argument, I guess. But I think like a tool like Text Attack kind of exists to expose that in a way that even though it might seem like it's human in terms of understanding, because it can exceed the human score and all these different benchmarks, if you use Text Attack or something like that, you can kind of gain some insight into why it might not actually understand anything at all. And so I'm not sure if deep learning is the full solution, but I think it's certainly part of it. And moving towards systems that have that type of, like, true understanding of language is really exciting and compelling to me.

That's a whole other episode that you're moving into right there. There's a. There's a lot of ground there. But I got to say, thank you so much for coming on the show. This was a fascinating conversation. It was a pleasure, pleasure to have you on the show, Jack, and it was a pleasure to learn about Text Attack. So we hope you'll come back sometime.

Yeah, thanks, Chris. I hope so too.

Thank you very much.

Thank you for listening to this episode of Practical AI. Hey, this was episode 99. You know what that means? Oh, yeah. 100 coming right up. Practically I is hosted by Daniel Whitenack and Chris Benson. It's produced by Jared Santa. That's me. And our music is brought to you by the mysterious one, Breakmaster Cylinder. We have awesome sponsors who support the show. Special thanks to Fastly, Linode and Rollbar for their continued support. Hey, did you know we take requests? We sure do. Head to changelot.com request, select Practical AI in the drop down and let us know what you'd like to hear about on the show. Pick a guest, pick a topic, you name it. We love to hear from you. Once again, that's changelog.com request. That's all we got for you this week. We'll talk to you again next time.

Changelock plus plus.