Modern NLP with spaCy

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by Digital Ocean, the simplest cloud platform out there and we're excited to share. They now offer dedicated virtual droplets and unlike standard droplets which use shared virtual CPU threads, their two performance plans general purp CPU optimized They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, cicd, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers or active front end web servers that need to be full duty CPU all day every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive starting at 40 bucks a month. Learn more and get started for free with a 100 credit at.co changelog again.co changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're PracticalAI FM and now onto the show.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I am joined as always by my co host Chris Benson, who is a principal AI strategist at Lockheed Martin. How are you doing Chris?

I am doing great. How's it going today Daniel?

It's going really good. Been a busy week. Had calls with teams in India and if you're listening, hello earlier in the week and that kind of got my schedule off so I'm a little bit tired. We'll see how the conversations go. I think it'll be all right.

I think this conversation is going to wake you up because we're talking about what I think of as your favorite topic in AI.

It definitely is. As you know, I'm always doing language related things, natural language related things and I'm really excited. I hope you are able to get some questions in today because I have all sorts of questions. I'll pause every once in a while to let you get one in.

Yeah, I'm guessing this is the last moment listeners are going to hear my voice.

But today we're joined by a couple of the core developers of Spacy and co founders of Explosion. And we're joined by Inez Montani and Matthew Hannibal. Welcome.

Hi.

Hey. Thanks.

Yeah, great to have you both on the podcast. Appreciate you taking time out of, I'm sure, the busy development of spacy to join us. Really excited to. About everything.

No problem. Glad to be here.

Yeah. I was telling you before the podcast, I recently got your latest spacy stickers and have them proudly on my laptop.

Oh, that's so cool.

Like, I'm still. I'm still actually waiting to really see them in the wild because we've sent this round. We've sent, like, over a thousand sticker packs, and so they're everywhere, and I'm, like, waiting for the day where no, I'm, like, sitting in a cafe and someone has my sticker and I'm like, oh, my God.

Yeah. So for our listeners who don't know, every once in a while, I don't know how many times you've done this, but you kind of just put out the call for anybody that wants stickers to send you some info and you'll send them stickers. And I saw it on Twitter and I was like, yeah, that's. I've got to get in on that right away. And they're really great stickers, too. There's a couple, like, related to, like, Prodigy and Data Annotation, but then there's some NLP tattoos, I would say. Is that the characterizes?

Yeah, Kind of like the old school tattoo style.

Yeah, yeah, yeah. Really nice design.

Yeah. Sort of like, I think of it as a sailor sort of style.

Yeah. I've been joking that, like, I don't know, how many retweets should I ask for to, like, get it actually tattooed on me? Because I'm at a level where I have enough tattoos that it kind of doesn't matter as much anymore. And I'm like, well, yeah, sure. I totally walk into.

Walking into.

Yeah.

I mean, your threshold would be far fewer retweets than me. Like, you know, it'd be the first tattoo for me. Yeah.

Please don't get a tattoo.

Like, it's so unique to. Not.

On that note, I would love to hear about both of your backgrounds outside of tattoos. Would you want to give a little bit of your background and then maybe Matthew as well?

Yeah, actually, I mean, we kind of need to start with Matt, actually, because that's kind of a better story. Or should I start maybe? Because.

Yeah, actually, I guess when you normally do the lead up, you just do this first. Yeah. So I'd been working on natural Language processing for a long time. I started my PhD in 2005 and graduated from that in like, you know, 2009, 2010. And then I was doing research on this for a few years after that as well as a postdoc. So basically like, as the technologies improved and there was more and more interest in this, I saw there were companies who were trying to use my research code. So, you know, I'd written some blog posts that had gotten some attention. And so, you know, I just had a GitHub repo sitting there and so people were trying to use this and I was like, well, it really was just supposed to print results and exit. That was like its mission in life. And you know, I designed it quite tightly around that like core task. So I saw that there was actually a need for, in the software ecosystem for more production ready stuff and things that could, you know, basically cross that gap. And so I was, you know, I was at about the level where I was supposed to be writing grant proposals, which wasn't really my thing. And so I decided, well, okay, if I leave this, I can have a go at starting a company and starting something with that. And then soon after this, met Ines and then we started working on, I think it was the display scanner.

Yeah, I think actually, yeah, because we, we met and I'd always done like. So I actually, I started making websites when I was a teenager, so that's kind of how I got into programming. And my degree is also partly linguistics. So I kind of had a good idea of what Matt was doing there and like, yeah, he always wanted to have a visualizer and have like better user experience. And at that time I was working as a freelancer and I remember like the first thing I actually said was, was like, look, I totally know what you're looking for there, but it sounds a bit boring. I don't know if I want to work on that. I have other things to do. So that was actually my first reaction, but I ended up doing it validation. So yeah, we did end up working together and we very quickly saw that there was like, I don't know, a lot we could do with both our skills kind of combined. And I started working on the core library spacey shortly after that and that was kind of when it was first released and.

Yeah.

Yeah. So when was that?

Early 2015. Okay, wow. So that's like, that's quite a long time ago.

Yeah, yeah.

And in terms of the initial ideas, did the company and sort of consulting things and other stuff like that come first or was the initial idea really to build the library.

So when I was thinking about leaving academia, I had, I guess, a, you know, a range of ideas for exactly what I could do. One of them was actually to work on software to assist language learners. So, you know, I saw that, well, okay, the tools for, you know, learning another language are kind of primitive and there's kind of a computational linguistic angle on that. Then I quickly saw, well, okay, it wasn't quite what I wanted to do, but, you know, and I saw, okay, there's, you know, a gap in the software ecosystem for a library like this. So it was very much around like, okay, well, there's a potential for having something there that's going to be useful to people in a commercial context. And I think that the way it would be most useful to people would be if it was open source. Because I feel like this type of technology, if it's closed source or if it's like under an API or something, it's just not as useful. And I thought, well, okay, if we can make the software useful to people, then there'll be a range of ways that we can support it commercially as well. Especially if we, you know, keep it relatively small and don't try to build it as like, you know, don't try to necessarily have a story for how it could be the biggest company in the world or something. There would be plenty of like, you know, interest from companies to, you know, make their usage of it a bit better and gain something and have a commercial opportunity around it.

So, yeah, so like, yeah, Spacey was definitely there first. And then when we started the company, that's when we thought about, okay, how are we going to make money? Or even we had ideas for products we wanted to build. We didn't want to take venture. So we were like, well, okay, we have users who want to use our stuff. And so we put out kind of a call for consulting and we had quite a few companies applying and that we ended up working with. And that's how we initially bootstrapped explosion when we first started. For like the first six months, I think we did consulting.

Yeah. And that was enough to get our first product developed. Prodigy, which is an annotation tool that, you know, is on a sort of old school software licensing where you pay for it and then you keep it and, you know, instead of like renting it like most software is these days. And that's been enough to keep the bills paid and lend some since. And the team's been growing slowly since then as well.

So I'VE got a question. I know Daniel has been intimately involved in using Spacy. I'm kind of curious as a newbie, though. Why is it called Spacy?

So initially, the very first idea that I had was around tokenization because I thought that the tools for data weren't really up to production grade. And it's the first thing that anybody ever needs to do in natural language is split the text into tokens. So I was like, well, it's based on spaces. It was short and it wasn't taken. And, you know, I had been working in Cython for a while and I liked, you know, basically developing it as a Cython program with like, you know, basically a Python API. And so that also, you know, emphasized the sort of speed aspect in the Cython aspect. So the same way things are like everything PI. This was ending in psi. And so I was like, okay, spacey, it works.

Yeah, yeah, definitely. And just for our listeners, we'll try to clarify, for those that aren't familiar with natural language stuff, some jargon throughout. So tokens and that sort of thing.

I'm going to keep you honest there because I'm the only person here who's not an NLP expert.

Yeah, so like tokenization and tokens, you mentioned, like, spaces. So you got text and you've got, you know, words in that text. And tokenization would be what?

Yeah, well, tokenization would be to split the text into words. And it sounds, if you just look at it, it sounds very simple, but it's actually once you get to things like punctuation, more complex ways of phrasing things, contractions. Yeah. In English, for example, that gets a lot more complicated. And then there are also different definitions. Okay, what's a token? Token is not always necessarily a word. Okay. In English you just have lots of like, small words that form bigger words. In German, for instance, a lot of these would be like one word. Or that's where all these jokes about the German language come from, that we have these like massive words that express something very, very specific.

Yeah, well, but, you know, in languages like English, we still have the same things of terminology. It's just the terminology is, you know, written with spaces in it. So, you know, natural language language processing is a term that we use and we call it that specific thing. And in German, that would be written without spaces, but it doesn't actually.

I do think there would be probably one space in it.

Oh, okay.

But it's more like income tax returns or something. That's definitely one word. In General.

Yeah, or federal income tax rebate guarantee or something. I don't know.

Yeah, I'm pretty sure that's one word.

Yeah, like you say, it creates all of these complexities that you don't realize. First, because in certain languages, certain things are said with a certain number of words. In other languages, it's not the same number. So people might, for example, think of translation as, you know, from one word to another. Just a sort of word to word thing. But it gets much more complicated than that quickly.

Yes, exactly. And these differences between language are sort of seen in the algorithms that have been developed in the way that people do things. So I often tell people that, you know, you can basically expect that any natural language processing technique will work best on a language depending on how similar it is to English. So English being the language that's most similar to English, everything works best on English. And it's not because there's like, I don't know, some magic property of English that makes it easier or more amenable to computation. It's just that, you know, for the last, like, you know, 50 years that people have been thinking about these problems, the dominant language that's been the test case that people have been developing towards has been English. And so that's the way that. And so you can really see that bias in the way that the algorithms have unfolded. And so even when algorithms touted as language independent, it's like, okay, the algorithm doesn't have any, you know, might not have any specific thing where you need to have a, you know, a resource that depends on a particular language. It'll still work better or worse depending on the characteristics of that language and the complexity of like an individual word versus, you know, how free the word order is. Like, all of these things will affect that.

So one of the things, before we go too much farther, I was wondering is kind of. I know you know, we're talking about spacy and I know you mentioned Prodigy. Is there anything else that explosion AI does or is it really focused on those?

We mostly are a developer tools company, so we definitely, we have a few other open source projects that are kind of cool. We have like some other projects in the pipeline and products that we're working on, but ultimately that's what we're doing. We're not doing consulting anymore. We haven't been doing consulting for a long time and we are building products for developers who are developing machine learning and AI systems.

Okay, gotcha. Yeah. And Prodigy, as you mentioned, is a data labeling system. Is that the best way to put it, I know that it's more than just a user interface. It actually integrates with models and other things, right?

Yeah, yeah, it's like a Python library. I mean, we sometimes refer to the whole concept as machine teaching because it's. Yeah, sometimes it's often a bit more than just like annotating or labeling. Because if you think of labeling, you think of, okay, you just get presented with something and you sign some label and that's it. Whereas, you know, Prodigy really lets you script much more complex workflows, try out ideas, iterate on the label schemes, really develop the models and how you want to structure your data. But yeah, essentially it's. You can download it, you can install it pip, install it just like any other package, and then it runs on your machine, on your data, just like, you know, like back in the old days when you would buy Photoshop and then you download Photoshop and then you have Photoshop and then you can keep using Photoshop.

We are chatting a bit before jumping on the podcast and you were mentioning that both of you are really passionate about the sort of workflow and production details of actually doing natural language processing machine learning in a practical setting. Is that kind of where Prodigy came about, that you were seeing that slowdown in terms of, I guess, machine teaching and iteration around models and all those things?

Yeah, so it's definitely informed by that. So we already had this pretty strong hunch that annotation tooling was a missing piece in a lot of people's workflows. And when we were doing the consulting, it really confirmed this as well. So the data annotation was really where people were lacking in the terms of the success of their projects. And if we were thinking about why somebody would fail at a natural language processing project, and indeed many of these projects are going to fail because it is exploratory and stuff. It would be around these data questions and around the questions of, like, okay, they couldn't get the annotation scheme such that they could carve up the problem correctly to make the models work well, or they didn't have the annotations at all, or they had a process that. Too slow, a feedback loop loop between the annotations and the developers. And all of these things were things which we wanted to address.

Yeah. For a long time. I mean, people were just like, okay, even though the data was the core of the application and really what the whole model depended on, they would like, just throw this on like Amazon Mechanical Turk with some vague instructions and I don't know Some survey that looked like early 2000s online survey and then wait to get the data back, get it back two weeks later, and then, yeah, it wasn't that great. That wasn't very surprising.

Yeah, a lot of effort for not much useful data, Right, Exactly.

And you know, it's sort of interesting to take a step back and think about the economic irrationalism of some of this stuff because the data science salaries are pretty sky high and they've been sky high for a long time. And now even the machine costs of running the experiments are quite high as well. And if you look at all of the like, you know, efficacy of those ingredients and how many thousands of dollars of investment are going into things, Data lab should actually be pretty cheap. And the tooling is pretty cheap as well. So for want of like, you know, $1,000 or $2,000 of annotation, people would be spending these, you know, tens of thousands of dollars on a project that, you know, basically will be doomed to fail. So you can spend way more computation and try to eke out like a longer hyper parameter search and get an extra 1%, or if you annotated like four times as much data for like about $4,000, maybe you'd get like an extra 12%, 10%. So it's pretty starkly, far more effective to be optimizing at that end of the scale.

Yeah. And so if that's what you're doing. Also very early on we saw, well, the problem is not the fact that we have to collect data. That's what people often, how people looked at this a lot like, okay, we have to find methods like unsupervised learning. How can we just not label anything anymore? It's like, well, labeling data is actually great because that's how you can program your model. Like you tell it what you wanted to predict. The problem is the tooling. Like, if, you know, if the tooling is bad, then yeah, okay, the data is going to be bad. And if we can improve the tooling and we can actually make it much more efficient and better to use, more productive, then we can also end up with better data, better models, a better data science process.

In particular, we saw that one of the, the problems that people had or one of the reasons why they hit this stumbling block on the annotation was that was often where people would hit the organizational boundaries. So that would be the part where, okay, you now needed to request a different type of budget, or you needed to cross over into a different type of basically organizational thing, or you'd have to have a meeting with an upper level manager because the annotation team was across an organizational boundary because they had some people who didn't have much else to do. So they became the annotation team. We sorted actually all of those things where now you've got an idea and you want to train a model. Now it's like, okay, but then I have to schedule a meeting for next Wednesday, pitch my idea and get buy in for this. And I'm not at all sure that this idea will actually succeed. So that's really where the innovation or iteration speed goes to die. And that's why we actually set it up to be much more built for rapid prototyping, so that data scientists could do some of this annotation themselves and say, all right, well if I've got an idea, I don't need to ask anybody's permission about this. I can do the annotation to test it out over, you know, between now and lunchtime. And you know, it's just sort of part of my daily work or like, you know, okay, I experimented with this and it's not something where you need to, you know, basically taking all of your hair brained schemes up to management and pitching them in meetings because that's not a productive way to work. Foreign.

This episode is brought to you by Brave. Big news from the Brave team. Version 1.0 is official. That means our favorite open source, privacy focused, blazing fast browser is ready for primetime. Their brand new iOS app landed just in time for the announcement and the Brave team is celebrating by granting 8 million basic attention tokens to the community. That means when you download the iOS app, you get 20 bat absolutely free. Put it to good use by heading to changelog.com hitting the triangle icon in the upper right hand corner and flipping us a tip.

So I'd love to switch directions just a little bit here. I saw one of your recent blog posts which was talking about state of the art NLP models and there's a sort of simple four step formula for a lot of these models and I think that might be a good way to maybe introduce people, people to state of the art nlp. And also I found it really useful myself to kind of have that scaffolding in my mind as I'm approaching these problems. So I was wondering if you could introduce that formula a bit for these state of the art NLP models.

Sure. So let me see. So I guess one way to think about it is these neural network models are all these sort of trainable functions that you can plug Together. So the details of exactly how they work sort of differ. And there's lots of different ideas for those individual components, but you can kind of take a step back and think about them as the input outputs of them. So one type of input that you'll have for language is, well, it's always going to kind of be a sequence of these discrete symbols. So at least if it's text data, right? So it'll either be a sequence of like, characters and, you know, then you'll have an ID per character, or it'll be a sequence of words. Or, you know, you could chunk it up into different segments that you've got different IDs, but it's going to come in as this stream of numeric identifiers. The first transform that you want to perform if you want to apply neural networks to this type of input is you need to somehow take that sequence of discrete IDs and map it into a dense representation. So vectors. The simplest way to do this is to just have sort of a lookup table where, let's say you've got a vocabulary of, I don't know, 10,000 words. It'll be a table of like, with 10,000 rows and say 300 dimensions. And then you'll take some word and it might be like, you know, the 50th most frequent word in your vocabulary. So you'll select like row 50 of that table, and then that embedding table will be the parameters of the model. And you'll sort of train this to have representations where similar words are sort of mapped to a similar meaning. So you'd hope that dog and puppy would have similar vectors and, I don't know, drink and eat will have similar vectors and it'll all kind of work out as this sort of vector space. So that's the first embed step.

Yeah. So basically, neural networks like to act or like to operate on numbers. Right. And so when you have these sequence of symbols or characters or words or whatever it is, in some way, you have to represent that in numbers, correct?

Sure. You know, in fact, every machine learning algorithm, neural or otherwise, is going to need to work on numbers in some way. The thing that was like, I guess, a puzzling challenge when I first started using neural networks as opposed to the other models which we were working with, is the other models really like having sparse representations, like, you know, you can have an ID that's sort of just a key in a dictionary, and it doesn't matter how many keys you have, or like, doesn't matter what the total space of the keys is you only care about which ones you happen to see in neural networks. It's not like that. You want to have a denser representation where you've got like some relationship between those. And that's kind of nice in that, you know, other each. Even if you haven't seen many examples of a puppy, I know that it's going to have a similar representation to this other word. So you've kind of got that sort of relationship between things in a denser representation. And that's one of the advantages of neural networks.

Yeah, I mean, ultimately, I think the main challenge is always there's so much knowledge about the world and the language that we kind of encode in text. And we want to be able to encode as much as possible about all of this extra knowledge in our model as well. And that's also, if you look at the recent developments in nlp, that's kind of what it all circles back to. And that's also, you know, the better the representations, the more we can do.

Actually, you already said the word I was about to ask about, and that was encoding. If you would go there when you're ready.

Right. So the next step, you know, if you've got this fourth step, is like embed encode. So after we've got like, you know, a vector representation for these word IDs, well, we're still at the stage where the vectors are kind of, you know, isolated. You've just looked up this word ID and you're going to have the same representation for that word type, regardless of its context. So every time I see a word like duck, I'll have the same representation, whether it's a noun or a verb or, you know, so I'm not getting any sense of its like, sense and context. So the next step that we, we want to do is sort of rewrite those vectors based on the words around them so that if it were based on the surrounding context, I can get another representation out. So this is a sort of input type that takes like, you know, this sequence of vectors and it outputs a sequence of vectors with, you know, taking into account the linear order of the vectors that were put in. So we take an unordered set and we output like an ordered set. Well, wait, that's not quite right. Basically we take the same sort of dimensions of like, input and output, but now we've encoded the context around it. So one way of doing this is to say, okay, I'll just concatenate the vectors for each word with the some of its surrounding context and I'll use that to recalculate that vector. So this is a convolutional operation, or you can have a. You can read them in turn and at each point remember something of the context before and rewrite the vector based on that. So that's something like a recurrent neural network. But regardless of which method you use to encode that context, we probably want something like this, because if we don't have something like that, then we're not going to be able to see the meaning of the sentence together. We'll have to just see the sentences, the sum of its words or something, which is not the way that language actually behaves.

Yeah, I noticed in the article you talk about both LSTMS and GRU architectures working well for that. Do you have a preference or is there one that you would recommend over the other?

So these days the transformer architecture is better than the LSTM for most situations, partly because it's just more amenable to the way the GPU performance characteristics work. So that's the one which pretty much everybody's using in place of the LSTM these days. Because we want spacy to work well in relatively small networks that you. You can use it without a gpu. We use a convolutional architecture instead. But I would say that those are basically the two architecture types that I use for the encoding step. The LSTMS are still sort of useful, but in most situations, especially for larger models, transformers work better.

So you mentioned the embed and encode. Apparently I've heard attention is all you need and you have a step that's attend. So maybe just. So I think the two steps left are attend and predict. Why is the attend step important?

So I should probably have called this reduce rather than attend. I guess I was kind of hinting at like the attention layers were getting a lot of attention at the time. And it really would have been more useful to call it reduce. So the input and output types of this are you take your matrix of things where you've got one row per word, and you basically want to get some sort of summary vector out of that. So you know, I want a representation for the whole sentence rather than just a representation that includes all of the words of the sentence. So the simplest type of this would be just summing them all up or taking the average of them or something. But you can also take basically more parametric view ways of doing this and have one of these attention layers or something like that. You can also use an LSTM for, for this as well and just take the output state. So there's again, a variety of ways of framing this, but it's essentially just a reduction operation where we take a matrix and we output a vector. And then finally the predict step is we take a vector and we want to get an ID out. So that's kind of the last type of thing that we want to do. So if we think of those as the sort of four data types of four like, signatures that we're going to have in these networks, well, we're probably going to be basically composing layers that look like that. And then there'll be other details for ones where you've got like, you know, two vectors as the import and then you want to attend over them. But mostly that's what we kind of want to do.

And then I guess in the end you have to predict something, right?

Sure.

So you've got the reduced vector, outputs a single vector, and then the prediction. Is it kind of the opposite of where you started with the embedding?

Actually, you can kind of think of it that way, but you can also just think of it kind of like a linear model. And you're just going to say, all right, take a weighted sum of this input vector and some weights, and at the end of it I'll like, you know, do some sort of max operation and say, okay, that's the one with the highest score. So it's just like other types of machines, learning models.

So I'm assuming that kind of getting back to where we started with spacy, I'm assuming that spacy can help you do some of the things that we've just talked about in this formula. But maybe there's other things as well. What's the sort of range of things that you can do with spacy? I guess?

Well, first, spacy is really a library that lets you put together a whole NLP pipeline of the different things you might want to do and extract from your text. So, you know, often that's like, you know, you're not just interested in predicting one thing you might want to read in your text. You want to find the individual sentences. You want to find out which concepts are mentioned in the text, like which person names, organizations, dates. And then you also maybe want to predict something about, like, what's in the text. And maybe later you want to relate these things that you've extracted to each other and compute something else. So the idea of spacy is you have a pipeline you can plug in functions into your pipeline. Some of them can be these machine learning models, others can just be a function where you can write some regular expressions, you can do whatever you need. And that's kind of the core principle of spacy. We always had our own implementations that usually have a good trade off of accuracy and speed, especially also on cpu. But you can also write your own models, plug them in, and then at the end of it, you can feed in your text and extract things from your text at a very large scale.

So I guess I'm curious, like, how would you implement a pre trained model? How does that fit into spacy as a component? And maybe contrast that with, if you were going to do it from scratch, how would you do that? How does that change the workflow for you?

So it depends on exactly what you mean by pre trained. Do you mean a model that's been trained for a particular task, or do you mean weights that have been initialized with some language that you can then sort of leverage the knowledge from them?

I think from my standpoint, I'm thinking of kind of doing transfer learning. And as the newbie in this group, if I was going to dive into it and I'm taking somebody else's model that I want to utilize for a particular task, and I'm wanting to stand on the shoulders of giants before me, how would I go about doing that as a newbie versus someone like Daniel, who's done tons and tons of work in this space, and maybe he's wanting to go in and do one from scratch. And how would it be different for me and Daniel?

Okay, so there's a number of different technologies around this. So, you know, most of your effort is still going to be around like, you know, creating the annotations for the specific problem that you want to do. And I would actually say that, you know, okay, you should sort of mostly be thinking about and focusing on getting the questions around that. Right? Because there's actually a surprising number of choices to be made in how you frame the annotation problems. So for instance, we have a number of users who want to work on medical text, right? And they say, okay, I want to recognize symptoms. And so then naively you'd say, well, okay, if this says, you know, patient suffers from asthma, that should be counted as like, you know, recognized as a symptom, and that should be highlighted. And then if I have another one where it says, you know, patient used to suffer from from asthma, then that shouldn't be recognized as a symptom. And so the sort of immediate Intuition is okay, that's the annotation scheme, that's what I should annotate. But that way of framing the problem will be vastly harder to recognize for the models because you're coupling the two pieces of information about mention of this thing and whether it actually was exhibited or not. And if you can find ways of framing the problems that you factor out those pieces of information, it's a lot easier. So I would say as a newbie starting out, that would actually be where most of the complexity lies. Then in terms of the actual software of using the pre training, Spacy has one facility which is just this command pre train. And you can either download the weight, some weights from us or use it yourself and that will, you can use that to initialize in Spacy train. And then we're also working on, you know, better support for transform models in library spaceytrain transformers that you can use to Burt and Excel Net models that have been trained on lots of text.

Yeah, or even if you're just starting out, even just like plain old word vectors like the common Coroll glove vectors have been trained on a lot of text. Even that can give you like a nice little boost and that's like super easy to use. You don't have to think much about how it interacts. You just initialize your model with that, then you write a little script that trains your model on your data and then hopefully you get some nice results out at the end.

So I think what you emphasized before in terms of NLP often being like a series of tasks that you want to string together often because there is so much pre processing and there is like multiple things that you might want to infer from text. And you mentioned a couple things like tokenization. You've mentioned finding certain things in the text which I think you're referring to like finding entities like people or organizations so like named entities in. In text. I was wondering, so those are kind of building blocks that you can put together in these pipelines. I was wondering if you could mention maybe just some of the, some of the most frequently used of those sorts of building blocks in Spacy. And I'd also be curious because I have my own perspective from different things that I've done. I was curious, as you view like the community using Spacy, have you been surprised by which ones of those things have been used most or people have found most useful? Or maybe it's the things that you thought they would find useful. I'd be interested to hear some of that.

Yeah. So I think definitely what People use the most is named entity recognition, as you just mentioned, and text classification, where you really predict one label over the whole text. We also, our default pipelines will also ship with a part of speech tagger, so you can predict whether a word is a verb or noun and the dependency parser, so you can predict the relationships between the words and for example, find out what's the subject and what's the object because that's also quite important. That's maybe one example where we think, oh, actually the dependency paths can in some cases be quite useful. But of course it's maybe not as popular as it used to be because we now have better ways of, of predicting these things end to end. But definitely it makes total sense to me why people would want to use named entity recognition and text classification because that's the most useful information you can extract. That also you can then translate into the business problem you're actually trying to solve. And yeah, and one other thing, the rule based entity recognition that was actually under matching, that's something I'm actually quite happy to see people use more. And that's actually very popular. So you can think of it kind of like regular expressions only that you can write rules that really take advantage of the token attributes and maybe even things that the model predicted. So you can say I want the word duck, but only if it's a noun and not a verb. And then I want to also extract an adjective if there is an adjective, otherwise not. And you can write rules like that and then also use that to extract more complex information and for many, many tasks. This, this is actually a really, really powerful tool and works much, much better than just trying to predict all of it end to end somehow. So it's nice to see people use these hybrid workflows of statistical models and rule based systems.

Yes, I'm so glad you mentioned that because I think it is often overlooked and especially for maybe cases where you don't have as much data, you know, lower resource data languages or something like that, you know, statistical and rule based approaches can be really powerful. I know recently we were trying to figure out how should we predict if a certain sentence is a question or a statement. And we looked into various things and we tried out various things like you know, text classification and like a larger model and all these things. But it ended up just some simple rules performed pretty much as good as any model we could train. So it was like I could see how it could be overlooked a lot. And I don't know if you see that.

Yeah, we Hear that a lot as well. Also sometimes it comes from within the organization. Like I often talk to people are like, oh, I need to extract digits for my text. How can I do that with the entity recognizer? And I'm like, actual digits. Well, you just write like regex, right? And they're like, no, no. My manager says we need to use like nlp, but that's not as cool. Yeah, we need to, we need to train a model. I'm like, God, that's like, you know, I'm sorry for your position in that company because that really sucks. But stuff like that, definitely. Or another thing we always try to tell people is this thing about, okay, build at least some rule based baseline that you looking to beat. For example, for your question task, you do want to find out how far do I get if I just check whether the last character is a question mark before you start predicting things?

Surprisingly a long way.

Yeah. And maybe you're like, whoa, that's 80% or 85%. You're like, well, okay, that's good to know because if your fancy machine learning model only gets 82% out of context, that would look pretty good. And maybe, you know, you could show this off and it's like, wow, that's great results. But if checking the last character gives you a higher accuracy, then yeah, that's probably not what you want to ship.

Yeah, this is, it's especially important with the deep neural networks because it used to be that, okay, you can pretty much rely on some of the other models on like sort of reverse engineering or getting the accuracy that you would get from the simple rules. But you can training a deep neural network, you're often sort of running blind and you have no idea whether the score is any good. And you can find that, okay, actually I'm dramatically underperforming like a bag of words baseline or something. And so part of the. It's very helpful to have this sort of existence proof of knowing where you should be because then you know like, okay, what to keep trying or when to keep trying and that sort of thing. So I definitely feel like having that sort of perspective on where you are or where you should be is very helpful.

So one of the things that I was wondering was I noticed that you talk about spacy, being designed intentionally to be blazingly fast. So it begs the question for me about kind of, if you're focusing on performance, what are some of your strategies for making spacy blazingly fast? And also because we're talking about performance, it also makes me wonder who are your target users on that versus like what I mean by that is, are you from a performance standpoint, are you thinking more about the data scientists that are creating the models? Are you thinking more about the engineers and the fact that for deployment and such? Because I also noticed that you talk about its production, really focused on production. Could you speak a little bit toward performance and target users for that performance?

Sure. So the things that are important for performance have kind of changed over time as the technologies have changed. So it used to be that the fact that it was basically implemented from the ground up in Scython was very important for some of the performance aspects because the actual maths that was being done in the model were kind of simple. So it was very important that all of the data structures were in basically memory managed code. Now that it's more around neural networks, some of those considerations are a bit different and there's a bit more forgiving and Python layer can be a little bit slower because there's kind of more maths that's being done that, you know, is kind of a slower bit that masks the performance of the other parts. So I would say in terms of the sort of target use case, a thing to keep in mind about natural language processing is that the problem sizes constantly get bigger. And this will continue for a wide range of companies and wide range of applications. So the working set of a problem that you're trying to handle will constantly accelerate. So let's say you're a news site or something, the volume of comments that you're processing or the number of articles in your archive, all of that's growing. And in many cases it's actually growing faster than Moore's law. So the sort of standard approach that people have for computational efficiency of, well, I'll not worry and it'll just kind of stop being a problem. It'll just kind of inflate away. Doesn't actually work so well for a lot of the problems that we want to do with natural language processing. If you want to work on the Twitter, firehose or other social media monitoring, again, the problem size gets bigger faster than computation gets cheaper. So we need to actually worry about the models being quick enough to work on those things. The other consideration is that if you make the models slow, then deploying them over a very large cluster is just. It's a hassle that's never going to get easy. Like the more instances you have to spin up, the just harder the problem gets. You get failures of the nodes you get. It's just Hard to be marshaling work over a very large number of workers. And so if we can make the models like you know, 50 or 100 times faster than the just operator expense of running things in practice gets a lot easier. And then finally there's latency. So there's a lot of applications where you care a lot about the time to response of one or two things because you want to have the model in the loop of some sort of user facing decision. And there again you need the models to be reasonably fast.

So I'm going to switch directions a little bit here maybe being that I am working for an organization that you know whose vision it is to see people flourish in community with the languages that they use most. I would probably get fired if I didn't ask about language support. So there's a lot of languages and I'm guessing that you know various of these building blocks that you've discussed have support in certain languages and not other languages. I was wondering if you could speak a little bit to I guess first the current language support, but also what people can do to expand the language support of spacy, how they can contribute, what's involved in that, what are the challenges of that?

Yeah, so I don't actually know the number now like I don't know how many languages do you support? Off the top of my head, I don't know, I don't know the number. But like it's in the docs but we actually the base support for a lot of languages is there and that usually just includes like okay, just some tokenization rules getting like basics. Right. But ultimately if you do anything for language and you want to train a model, that's kind of where the bottleneck is. And yeah, you can optimize algorithms for different languages but ultimately it comes down to the data and being like a library that's used in production. We are somewhat limited to like okay, we need to source data. We need to source data sets that can be used commercially that exist in the first place and that we can maybe pre train models with or that our users can use. And that's kind of what we're currently seeing is like the biggest problem. And that's kind of, that's not like anyone's fault directly but like that's kind of something we have to work with. And that's also why it's not so easy to if people are like oh, why don't you support. Yeah, why don't you give me a perfect pre trained model model for insert language here? Or like why does it? Why can't you support this? Or why is this language so bad? And it's like, well, we have to work with what's there. We have to, you know, we can run our own annotation projects, we can run our own data collection processes. But like that's, that's really the main thing it comes down to.

So have you seen contributions from various language communities around the world that really, you know, take ownership and contribute some of those models and the rules and all that stuff?

Yeah, so we've had a variety of contributions. So one that was particularly end to end and very effective was Yanis Darius did a Google Summer of code project where he contributed Greek support. We've had a number of people working on Indonesian. I think we had people working on Tamil.

Yeah, we had like, you know, with some more customers. Yeah, especially custom work. We definitely had a few users who went and used Prodigy to create their own NER annotations because that's also, you know, something that's usually lacking. We can have tree banks for dependency parsing and part of speech tagging, but like named entity recognition is like much more important to many users and also there's not enough data we've had. What else do we have? So actually yeah, I think some of the Nordic languages they actually, I think Norwegian, they have like good, I think government sponsored initiatives and have published good corpora under public domain and that's of course like incredibly helpful and also high quality data. And then we've had users from the community who saw that, ran some experiments on it, got like pretty good results, shared that with the community and that meant we were able to ship a Norwegian spacy model, for example, like a base model that people can build on top of the.

That's awesome.

Another one that's very notable is the Japanese support for a long time has been driven by Paul O' Leary McCann who's now doing freelance work. So if anybody's listening and they want to work on Japanese projects and they need assistance with this, he's a great person to get in touch with and you can email me for the contact for him. There's also.

He's been really driving all the spacey in Japanese stuff.

There's another group in Japan who've been working with Spacy and they've got their own library for this as well. So there's a number of contributions around that I would say for something like the perspective of sil. If you're thinking about the vast number of languages in the world, well, tools like Spacy are kind of designed around different use cases. Like, spacy is actually design for written text processing. And so for a great number of languages, it's a little bit putting the cart before the horse because you say, okay, I can make this tool that can process lots of text, but what written text do I actually have to process? So if you're in a situation where the language actually doesn't have much written text, then okay, it'll be quite difficult to get spacy running with things. But on the other hand, also, spacy wouldn't really be sort of solving a useful problem for you either. And so I'd say that's actually the position of most languages. Right. The other thing is that for those languages which are sort of less typically written, well, at least the writing system tends to have been designed by linguists. And so they tend to be easier to tokenize than languages with slightly more complex histories around their writing system.

Sure, yeah. And, yeah, I mean, there's a lot of great efforts. I know the Masakane project right now in Africa is working on NLP tools for a lot of African languages and trying to. I think it's also the Zendi effort there is trying to gather a bunch of data that would be relevant to this so that you would be able to start out and build something useful. But yeah, it's great to hear that you've had community contributions around that and there's kind of thriving communities of people that are wanting to help build in. Build in that support.

Yeah. And it's also, I mean, this is also part of the reason we really want to focus on, like, the tooling around, like, creating the data. Because, you know, it's one thing to talk about, like, oh, we don't have enough data, but like, if you, if we can have more efficient ways to script workflows that even maybe a researcher can say, hey, I don't have any labeled examples, but I'll just create my own little set so I can run experiments and like, move, get this moving forward. That's like, that's pretty good. And I think can have a big impact if. If you don't see annotation as this huge crowdsourced effort and something focused that is actually quite easy to achieve.

So it's interesting for me, as Daniel just made those comments about some of those efforts, and I'm kind of wondering, as someone relatively new to this aspect and new to NLP compared to you guys, can you give us some perspective on general trends in natural language processing and what are the exciting things that you see happening going forward? You know, what, what are you excited about over the next couple of years?

So I think it's definitely exciting to see the field develop and to see so many more developers get skilled up with this. So I would say that, okay, one way to answer this is if you went back a few years, what were the questions which people would asking were asking about how trends would develop or like what would happen. And then if you compare to to now, I think you see an interesting perspective on that. So one of the things that people were wondering a few years ago was whether people would be running the models themselves on their computers or whether everybody would just use an API. So would everybody use the one API that was like the Google Understand Language API or something and that would just be what people would use for this or would it be the case that as we've seen now, actually more and more people are building the models and more and more people are involved with quite detailed libraries and have even switched over from a library like tensorflow to something more flexible like Pytorch. And so I think that's been answered decisively in the direction of, okay, people want it to be programmable and they want to understand the workings of the model. They don't want a black box that, you know, where everything's done for them. And I think that the reason for that is that there is no sort of one answer for a lot of these things. You need to score script the problem yourself. You need to have it recognize exactly what you want it to recognize and the model to work the way that you want it to work with the features that you want it to work with. And so I would say that that's definitely been a trend that we expect to continue and we expect the general sort of savviness and knowledge of people and they'll want to work with basically the most effective ways of doing this rather than the ways which are like superficially the simplest.

Yeah, yeah. And I think also even if you look at like new developments like transfer learning, that's of course very exciting because, you know, it means we can reuse knowledge better and transfer it between different things we're training. And so we do think that actually, you know, there's some trend that it moves away from this idea of like, oh, we need this big, big, big data, huge operation and we can actually work on like kind of a medium scale, try out a lot more things in the workflows. And also I do think, yeah, we see a lot of systems that like, just work end to end. And you know, people are like, well, cool. If I can just like throw like Bert at it, it'll just magically work. But I think as the field develops and also as the problem develops, I do think there's still a lot of challenges are much more on a level of, okay, I can predict all kinds of things, I can do it very quickly without needing like too much data. But how do I really translate these predictions into my very specific domain specific problem that I have to solve for my business use case? And I don't think there's like an easy answer to this on the technology level. That's like something you need you as an expert, you need to know what questions do I ask, what can I train, what will work once it works, how do I interpret the results and how do I all put it all together to answer the questions? And yeah, and that's something I think. Yeah, you can't really predict end to end.

So let's say that, and I hope that we have. But let's say that we've inspired some listeners out there to get hands on with nlp, with Spacy and with Prodigy and with the other tools that you're releasing. I know that you've actually built a course for NLP with Spacy, is that right?

Yep. Yeah, it's free, it's online and it's available at course Spacy IO and it's kind of interactive. You can, you know, you get some little prompts, you can enter code, you can check it. But it's like all. Yeah, intended for self study but it's been very popular and well received. So.

Yeah, yeah.

Would you, would you recommend that as kind of a good place for someone who's maybe toyed around at least with, with Python and done maybe some, some Scikit, learn stuff or something like that and then want to do something NLP related? Would that be a good place to start or their better place?

I think of course. I really try to design it in a way that it also explains all the concepts about nlp. So even if you've never worked with NLP or machine learning before, I think it still gives a good perspective and gets you productive. But also if you have done a lot of machine learning and no nlp, I think there's still enough in there that it feels valuable.

Yep.

Awesome.

And I would say that in terms of other resources for more general machine learning stuff or understanding neural networks a bit better, the fast AI course is very good. And then actually a little book that I recommend to some people is Machine Learning Yearning by Andrew Ng. So I don't like all of his talking points, but that little book is actually quite a useful primer in setting up machine learning projects. And it has some advice that sort of gets lost along the way about what's an evaluation set, how do we do these things, that sort of stuff. And it's a pretty, pretty short read, so that's also a nice background as well.

That's great. We'll definitely put the link to the course and to the book Fast AI. Again, I think we've linked Fast AI I don't know how many times at this point, Chris, but we're big fans. Yeah, we're big fans as well. And actually they have another course that's specifically NLP focused nowadays.

Yeah, no, that's actually. And I really liked. I kind of like the curriculum there because, because it also starts with the basics. I think it looks at like very basic rule based approaches, the whole history. And it's not just like, oh here's like you know, the hippest thing. And I think also it does cover a lot of ethical aspects too and bias in models which yeah, is also something you don't typically get from like your average programming course.

Yeah, sure, yeah. Really appreciate what that community is doing and what it's all the tooling and the courses and everything has meant through the recent years. Well, thank you both for taking time to talk with us. We'll definitely link everything that we've talked about in our show notes. There's a lot to explore. There's a lot of questions that I'm sure people might have. Feel free to join us on our Slack community. You can find that@changelog.com community or LinkedIn or Twitter or wherever you find yourself and get plugged in with the Spacy community. Try out some things and really appreciate you both both being here and looking forward to great things from Explosion and Spacey in in the future.

Thanks. Thanks.

All right, thank you for tuning into this episode of Practical AI. If you enjoyed this show, do us a favor, go on itunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend. Whatever you gotta do. Share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by fastly Learn more. Com and we catch our errors before our users do here at Changelog because of rollbar. Check them out@robot.com changelog and we're hosted on Linode Cloud Servers. Head to linode.com changelog. Check them out. Support this show this episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder and you can find more shows just like this@changelow.com when you go there, pop in your email address us get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.