OpenAI's new "dangerous" GPT-2 language model

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by Linode, our cloud server of choice and we're excited to share. They've recently launched dedicated CPU instances. If you have build boxes, cicd, video encoding, machine learning, game servers, databases, data mining or application servers that be full duty 100 cpu all day every day, then check out Linode's dedicated CPU instances. These instances are fully dedicated and shared with no one else so there's no CPU steal or competing for these resources with other Linodes. Pricing is very competitive and starts out at 30 bucks a month. Learn more and get started at linode.com changelog again lino.com changelog Foreign. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're at Practical AIFM and now onto the show though.

Welcome to another fully connected episode where Daniel and I keep you connected with everything that's going on in the AI community. We'll take some time to discuss the latest AI news and we'll dig into some learning resources to help you level up on your machine learning game. I am your co host Chris Benson. I'm Chief AI Strategist at Lockheed Martin RMS APA Innovations and with me is my co host Daniel Whitenack who is a data scientist with with SIL International. How's it going Daniel?

It's going great. Welcome back from Switzerland. So as our listeners will know, Chris has been recording Practical AI episodes on the road at Applied Machine Learning Days and really been enjoying those. But glad to have you back from your trip.

It was good to be back. It was a great trip. I met a lot of really interesting people and obviously recorded some good episodes. I did my talk just for the listeners. Daniel ran the AI for Good track remotely from America and it actually went off without a hitch. Everybody felt I was talking to the other speakers. There was no problem. So thank you very much Daniel for managing that from thousands of miles away.

Yeah, that was kind of an interesting experience. I had planned to be there, but glad to hear that. I was hoping that all AI people would just kind of converge there as expected and Sounds like that's what happened. So if you don't know about applied machine learning days, definitely check that out. It's a great, a great conference, and we've had recently some, some guests on talking about AI for good and other things. And that's, that's been really awesome. But now that we're here together again, we have a fully connected episode and I'm really excited today. Of course, if, you know, if you haven't been hiding under a rock and you follow AI stuff, then pretty much all you've been hearing about for, you know, a couple weeks or however long it's been, is OpenAI's recent language model that they've released called GPT2. We're going to kind of talk through some of that stuff today because it's, it's pretty interesting. Have you been seeing that online, Chris?

Yeah, it's hard to miss. Especially, you know, like, I think the very first thing I saw was Elon Musk's tweet about, you know, we have a model. I forget, I'm not quoting, but something about we have a model that's so amazingly good that it's dangerous and thus we, we have to, to not release the whole thing. And, and obviously that, like, like everybody else on the planet, that piqued my interest and started diving into it. And it's, you know, technically, it's fascinating what they've done. And, and then there's some pretty interesting ways that they've chosen to not only approach the model, but approach the, the release of it. You know, a little bit of drama around it here.

Yeah, definitely. I, I, you know, I've seen things that are like, of course people have been kind of captivated because one of the things that they're doing with this model is text generation, which we'll kind of talk through in a second. But the quality of it is just astounding, really. And people have been posting, like, different things, like they've generated, you know, reviews for their book or like, various stories and other things, and they're kind of entertaining, but all of them are pretty astounding in the quality of the text generation, which also, of course, leads a lot of people to be concerned because, you know, how do we know if this text has been generated by an AI or not? And what are the implications of that? And so, you know, Wired had this article about, like, you know, the AI that was too dangerous to release based on, like you were saying, some of what, what Musk and OpenAI has talked about. So it's really it's really been an interesting discussion. I don't know, I've seen some people kind of get frustrated with, with all of this talk about the danger of a. Which we can get into a little bit later. But what's your general feeling about this discussion kind of generally, Chris? Is it positive, negative, a bit of both?

I think it is the reality that we are moving into either way. So regardless of how you spin it or how you perceive it, we are in a moment here where we're seeing this GPT2 model that is able to. To make people believe that what the text has generated is indistinguishable from humans. If you, you know, they put the text in front of a number of people. And then on top of that, just as a side thing not to get into right now, there's been the, you know, all the facial stuff that I was also seeing in the news over the last couple of weeks, where there's the website where you can just hit refresh over and over again and a new person that does not exist in real life is generated by a Gan. And the reason I mentioned that is we're just moving into a moment where it is now entirely practical for these AI models to be able to generate things that are indistinguishable from the reality that we are otherwise in. So I guess to kick things off, do you want to maybe even back up just a little bit before we dive in and kind of talk about what a language model is?

Yeah, sure. So this GPT2 model, which is what they're calling it, which is it's building on a previous model which you might have been guessed was called GPT. But this model, along with a variety of other models that have been released recently, so those being like Bert or Elmo. So we had another episode, episode 22, where we kind of dove into a particular implement implementation, bert. So if, if you're wanting to know in a little bit more detail, like what a language model is and how to utilize it, you might listen to that episode, episode 22 about Bert. But any of these models, including GPT2, is really, when they say it's a language model, this is really like a pre trained encoder. And what that means is you kind of put words in and then out the other end comes these word embeddings or these various representations of the words that are based on kind of contextual relationships between all the words in your corpus. So these embeddings come out and then you can utilize those generated embeddings. For various tasks like sentiment analysis or named entity recognition and like question answering, text generation, machine translation. And so the language model, part of these is that, you know, encoding bit.

Yeah. And this is a particularly big one they describe. GPT2 is a large transformer based language model with 1.5 billion parameters and trained on a data set of 8 million web pages. Its objective is simply to predict the next word.

Yeah.

Huge scale though.

I'd be interested, like just, you know, as you were talking about that, I'd be interested to like, how did they parse and format these web pages as we'll talk about later? They, they didn't release the full data set that they used for this, so we'll talk about that later. But just, I don't know, thinking about how this would operationally work in my mind, you know, parsing these web pages is a little bit complicated in and of itself. Yeah, I don't know, it seems complicated and I guess 1.5 billion parameters, it's no small potatoes.

No, I think it's pretty huge. Yeah. I mean, and there's certainly the drama associated with it. They note on their blog post, due to our concerns about malicious applications of the technology, we are not releasing the train model. And then they go on to say that they'll release a much smaller model for researchers to experiment with as well as the technical paper.

Yeah, cue the ominous music.

I know my first impression when I read that was on the assumption that this model is as great as it looks like it may be here. You know, isn't that sort of, you know, you have a dam that's about to burst, you know, where it's just, you know, suddenly we have this new capability. Isn't that like sticking your fingers and little holes in the dam to try to keep the whole thing from coming? Because if it is, they think it is, and they're releasing this, it won't be long before it's pretty much everywhere because now that everyone knows you can do it, it'll be recreated elsewhere.

Yeah. And I think it should be noted that this really algorithmically, there's not really a major advance kind of in the architecture or algorithm that is the focus of this model, but it's really kind of the scaling up of it. So as you mentioned, Chris, this is a transformer based model. And so the other transformer based models recently have been, as we mentioned, Bert and Elmo and these things. And the transformer architecture has been around for a bit. So that's like this mechanism that kind of learns the contextual relationships between words or subwords in a text. And so that's been around. So that's not new to this GPT2 model. So that's not the new thing. The new thing isn't really how they train it because they're really just using this simple framework of training. So when you're training these language models, you need to have some sort of task that you're trying to do. Even though the goal is to get the embedder, the embedding layer, it's not to do classification or translation or something. You need some simple task to train the embedding on. And they're just using a simple task. So it's just predicting, like you said, predicting the next word in text from this Internet text. And so the task isn't really new, the transformer idea isn't really new. It's really the scale of what they're doing. So they trained it on this hugely diverse Internet data set or data set of web pages. And because of the diversity of that data, there's really some kind of significant capabilities that come out of it. Have you seen kind of this broad set of capabilities that they're proposing?

I have, and as I've read through the various articles on, looks like kind of going back to what you were saying, that the key differential in this is just scale. They put a lot more hyper parameters into it. They had a much larger data set, but they explicitly said they weren't really covering any new ground algorithmically. So it makes you, you know, as we're all starting to scale up over time, it really makes me wonder, you know, as fast as this is moving right now, if we're not going to be charging for it even farther. I mean, this, this was essentially the racetrack flag, you know, that went around and, you know, it's, it's, it's go for it. So I think we're going to see this, I think this is going to be so common within a few months out there that you'll see it in production pretty quick, regardless of the fact that they held back the larger model in this particular case.

Yeah. So maybe one thing we kind of want to pause and define. So you'll see as you kind of read through some of these blog posts and everything, they talk about like zero shot something and like multitask or like various tasks associated with the model. So have you encountered this idea of zero shot before, Chris?

Nope, this was a new one to me. So you want to, you want to jump in and explain?

Yeah. So the kind of general idea is that zero shot basically means that the model is not trained on data that's specific to a task, but you evaluate that model on the particular task. So let's say where I've seen this in the past is like in translation, if you say, have a model that translates English to French and then English to Spanish, you could train that model and then you could try a sort of zero shot thing where you translated not from English to anything, but you could translate maybe from French to Spanish. So the model wasn't trained on that data, but you could try it out to see how well it worked to do that task. Right. And so this is kind of the idea of zero shot. And what's really interesting with this model, I think what people are getting really excited about is that they train this model on this large set of data with a simple task, but it's showing really great results. I mean, not like state of the art, but good results for things that it wasn't trained to do. So, for example, text summarization, translation, question answering, these sorts of tasks where they're showing these zero shot results for things that the model wasn't trained to do. Which is kind of a crazy idea when you think about it.

Yeah. So what do you think the implications are for zero shot on training for the industry at large? So now that this announcement came out out, and people are diving in and you're going to see more and more in the weeks ahead and you know, is zero shot in this unsupervised approach, do you think that's going to be kind of the standard way that people tackle this going forward, given the result that we have initially here?

So I think that there's kind of two elements to this which are kind of sufficient data size and diversity and compute. So I think what they've shown is not that like these unsupervised techniques and, you know, generalization of a model to all of these tasks is like something that always can be done. But specifically they've shown that because their data set exhibits all of these very diverse kind of qualities. So there's like data about different languages. Right. And there's data maybe from question answer or forum websites or something. Because there's this sort of diverse set of data, it naturally encodes codes what you need for various tasks like question answering and translation and things. And so given that sufficient amount and diversity of data and the actual compute that you would need to train 1.5 billion parameters, then yeah, sure, this might be a good, really great starting point for a whole variety of tasks. I think the main issue here is not everybody has that diverse data and not everybody has that compute. I've never trained a model with 1.5 billion parameters. I don't know about you.

No, no. That's a little bit bigger than I've dealt with for sure. So I guess. But over time, as you know, we've been on this exponential curve with compute increasing. While you pointed out early in this episode that we didn't know how they were parsing the web pages, you know, they clearly took a data set that is publicly available to everybody. So we, we do have access to that if we're willing to put the infrastructure behind the collection and the parsing and the compute is becoming more. And it's really fascinating to me to start thinking about what the implications, you know, on all of our lives are going to be. It's really a science fictiony kind of kind of idea that's kind of upon us in very short order here. And so, you know, going back to what it's kind of funny so you and I are always talking about how people are concerned about the potential dangers of AI and whether they go in my current job with Lockheed Martin and it's actually become part of my job to be thinking about those types of things in the frame of conflict, obviously. And so, you know, one of the things as I was reading this, I was thinking about is if you go back and look at what Gans are able to do now and you combine it with this then and you think about, you know, all the, we've been talking about political misinformation over the last few years with, you know, various elections and stuff. I just wonder that, you know, that's the downside to it. There's also some pretty amazing upsides in terms of being able to create user experiences around these new technologies that can do some pretty, pretty wondrous things. If you combine, you know, in the medical industry, if you want to have a. Beyond just a chatbot, but essentially a virtual doctor who looks and talks very much like a real person, you'd never know the difference. And you're in a remote part of Africa. We've talked about, you know, being in places where you don't have ubiquitous Internet everywhere. It just, I think this is a real game changing technology that in tandem with these other game changers is really accelerating what we're going to experience over the next few years. I think the idea of the distant future is really upon us, whether it be good or bad. Any thoughts on that?

Yeah, and I Think maybe one thing that we can share just to kind of emphasize these sorts of implications, and really we can talk next about the dangerous implications of this, which really have to do with what they're saying around, like, fake news generation and that sort of thing. So one of the things that I think we do to drive that home is just read a little excerpt of, you know, some of this generated text, which is really just astounding. So this is kind of a silly subject which, you know, maybe people don't find interesting or wouldn't think is real, but imagine that this was kind of a real news story. So in one of their examples that they post online and one of their samples from OpenAI, they have a system prompt. So this is text that was generated by a human. And then they follow that on with kind of a model completed or a model generated text that actually just generates the rest of the story. So this first bit I'm going to read is the human written part. So they say. In a shocking finding, scientists discovered a herd of unicorns living in a remote, previously unexplored valley in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English. Okay, so that was the human written portion. And that's all that they gave to the model. And then the model generated the following completion. So this is all model generated. So not human generated. The model came back with. The scientists named the population after their distinctive horn, Ovid's Unicorn. These four horns, silver white unicorns, were previously unknown to science. Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved. Dr. Jorge Perez, an evolutionary biologist from the University of La Paz, and several companions were exploring the Andes Mountains when they found a small valley with no other animals or humans. Perez notice, blah, blah, blah, blah, blah. And it keeps going on. So you can already get a sense that, like, if I was to read and it kind of drifts in and out as the story goes along. But if just reading kind of that initial bit, I would have absolutely never expected a computer to be able to generate something that coherent, especially when it's been trained on only a very simple task. I don't know, what are your thoughts on unicorns? Chris?

Well, I like unicorns. Just to go on the record. No, I think when you're. And I'm looking at the rest, the rest of the text that you were starting to read through and the thing that jumps out is that it is so sophisticated in the way it's using language, it has sophistication of a well educated person as they might speak in a storytelling mode. And that's very different from many of the computer generated texts we've seen over the years prior to this. And it's that sense of sophistication that jumps off the page and you know, it's pretty astounding if someone wants to get on their blog at Open IA and read through the rest of the text. I mean, easily you could believe that all of this was written by a person. You might even challenge that it was computer generated. I'll say whatever. It's probably going to really change gaming going forward. Dungeons and Dragons will never be the same again the way this is. But I can't stop thinking about all the uses for this that we can apply in industry.

Yeah. So we've got to the point where we can see generally what this GP2, a GPT2 model is. They should make an easier pronounceable name and like the quality of the text generation that it can produce. So we've seen this kind of very coherent, sophisticated text that's generated by this model, which, you know, is just astounding. And so naturally, as you kind of think, as you were saying, Chris, there's a ton of great applications to this and maybe fun applications like you were saying in gaming, maybe really good applications in like text summarization or question answering and that sort of things, that sort of thing. But it naturally brings us to the point of talking about, hey, there's some really malicious applications of this as well, especially if we talk about, you know, fake news generation. So if you're able to generate, you know, basically endless news stories that are coherent along a particular viewpoint or promoting a particular viewpoint or idea or story that's fake, obviously that flood of really coherent fake news is definitely of concern. So you know, you were talking about in terms of security and all of that. You always are thinking along these lines these days, Chris, what's your thought on that sort of line of application?

When I was in Switzerland for the Applied Machine Learning Days conference, I also had a conversation with an expert on AI safety and he was working on models that addressed some of the very things that you were just talking about. And I guess seeing this come out, it's something that we've been discussing, but it made me realize how critically important the field of AI safety is going to be. I think just like we've been talking about ethics over the past year is crucial to this. I think different forms of AI safety in terms of being able to differentiate, you know, between what is fake. And what is not fake is going to be so crucial for not just the technology, but for society going forward. I think we're going to spend the next year talking a lot about AI safety as, because, you know, the genie's out of the bottle on this and whether we're worrying about the good or the bad, it's an amazing new technology. But we now have to be able to start being able to distinguish what that is. If you're, if you're talking about fake news and the ability to scale up on that, you know, on a downside, you could have just be awash in fake news and suddenly AI safety is all about where's the real news in that? If you're talking about a situation where you're in a conflict between, you know, between two nations or something, it becomes a weapon of war. You have to start having tools to distinguish between them. Those are both dark things. But, you know, that's, there are, there are people in the world that will certainly try to use it for malicious purpose, as pointed out in the blog.

Yeah, the whole idea that the dangerous bit of AI is, you know, AI is gaining consciousness and taking over the world, I think we can just put that aside for a long time.

It's kind of irrelevant.

Yeah. The most, the danger that you can see with this application of text generation. So humans can do a lot of things. Text generation is one of those. And even if we just see like this model is capable of the quality of this text generation and nothing else, that in and of itself has huge security concerns. So I can read the, the quote from the OpenAI release blog post that you referenced before. They say, you know, due to our concerns about malicious applications of the technology, they're only releasing a much smaller model and they don't release the data. They have a technical paper, they reference certain, certain particular malicious uses of it. So they list off generating misleading news articles. So that's what we've talked about. They mention impersonating others online, automating the production of abusive or faked content to post on social media and automating the production of spam or phishing content. So I don't know if it's good or bad that they listed out that. I guess people would have figured out those anyway.

There's a, there's a point that it makes and it goes to what you were just saying a moment ago, and that is that you, for many years the concern about AI going amok has been in, you know, what happens if AI becomes conscious or self aware and able to take actions that we were not anticipating. The reality is I'm unaware of anyone making any substantial progress down that road. But, but the thing that has mattered a great deal is that you have these tools, these AI tools. In this case we're talking about this GPT2 model that has nothing to do with self awareness. It's not self aware, it's not conscious, but it is so very good at one specific type of task that it is able to match or exceed human capability in its nearer thing. We've seen that outside the NLP space as well like the gans we were talking about bit a little, little while ago. And so you know, the concern, the danger, the ethical issues, the safety issues that we may be considering in going forward, I don't think is about consciousness. I don't think it's about the Terminator robot that's loose upon the world and going around killing everybody. I think it's about the way humans are applying these very specific tools that are just marvelous at what they do and they can be used for great good or terrible bad, terrible evil. And so I think that's where the real conversation is going forward is how do we want to do that. I think probably I'm guessing you are too. I'm kind of tired of people talking about Terminator robots coming to kill us because I just haven't seen that in reality. But this is a big concern here about how do we move forward into a future where these tools become commonplace.

Yeah. And I do appreciate OpenAI and Google and others, particularly OpenAI in this case being transparent about their concerns with this. I've heard, I've heard certain people say, oh well they're just like saying these dangerous things about danger because they want more publicity which you know, who knows what their, what their full motivations are. I, based on my perception, I don't think that that's totally it. Maybe, maybe there's a part of it that's that way. But I do appreciate their, their transparency around the, that they've thought through this. They've decided to deal with this issue by still releasing the research, so publishing the paper, still releasing the code, but only releasing a smaller pre trained model so not the full model and then also not releasing the full data set that they've parsed associated with this. So I guess their sort of thought process is oh well, people don't have the compute that we have. It would take them an enormous amount of time to recreate this data set and train the larger model on this data set. And so they're thinking, oh, well, this at least, you know, buys us time.

Exactly.

Right, yeah. Do you. Do you kind of track with that train of thought or does that seem not sufficient to you or just not relevant or what?

No.

What is your thought as far as how they've been dealt with the issue from their perspective?

Yeah. There isn't really a guidebook on how, how responsible disclosure is, is to be done in this. You know, different organizations have different approaches. I give them the benefit of the doubt that this, that they're trying to be responsible. It certainly doesn't stop the potential for malicious actors to take advantage of this. But what it does is it slows it down over time for the exact reasons that you just said. And it gives us time to think our way through it a little bit, which I think is good because it's still out there, it's still coming. We now know it's possible. And that means that everybody will be very much focused on it. It's already proven to work, and therefore there will be money behind it and there'll be interest behind it. So I think it really comes down to the fact that as we go forward, just as we have been talking ethics and as we are now talking AI safety, we need to build some frameworks around what it means to discover these tools, produce these tools, and release them into the public. I think they're coming. I don't think they're likely to be to stop at any point, but I like the fact that they're thinking, let's put the brakes on just a little bit to have time to react a little bit better than we can in the moment.

So I agree with you. I think pretty much in everything you said, there is one aspect of this that I don't know that I fully formed an opinion on, in the sense that OpenAI is essentially saying that they've judged this to have negative consequences in however they're quantifying that. And so they have deemed that it matters that they don't release things, rather than releasing things and then having the community like form be able to test it, be able to actually use it to, to, you know, come up with methods that would, that would fight against the negative consequences that it might produce. They're pretty much restricting it to themselves. And in that sense, other people can't really fully parse the consequences because they don't have access to the full thing. I've seen this argument out there, essentially, that OpenAI is. They're making this decision about it, and people said there's no excuse for waiting to release it and that sort of thing which, you know, I kind of get their train of thought. I don't know that I. That I fully agree with it.

I didn't mean to interrupt. I was just going to say I think that if you'll think back to recent history where we spent so much of the past year talking about the ethics of around AI and we've had experts like Susan Ettlinger on the show to discuss in that time period, as that conversation was being had within the community, you had a lot of the big players such as Google and Microsoft and others help by releasing. They thought their way through their own ethical framework and they released those guidelines that they were using internally. And those of us who have been the beneficiaries of that have been able to kind of form what we think around several different frameworks and combine and make something that we hopefully think works for ourselves. Maybe that's something we can do here from a safety standpoint is with this kind of release and, and presumably others to come as well, it gives us a chance as a community to react a little bit about how we want to frame this from a safety standpoint in terms of release, think our way through it a little bit and then do it. So maybe a year from now, two years from now, there's more of a standard way of doing it instead of kind of feeling your way on your own. So I don't really hold the carefulness of what they're doing against them.

Yeah, that's a good point. I think we'll reference that link to Susan's show and others in the show in the show notes. I would also encourage you. I mean this is a active topic of discussion within the community and we would love to really hear from all of our listeners what your thoughts are on this pretty controversial subject. We have a Slack channel, a practical AI Slack channel. If you go to changelog.com community you can join our Slack channel. We also have a LinkedIn LinkedIn group where you can make comments and so join one of those communities and let us know what you're thinking. If you have references to other good articles or other top guests that you think might shed some light on this, we'd love to have them on the show and we'd love to share those links via the the news feed@changelog.com so definitely get involved with that and the show notes.

By the way, while you're mentioning community, the show notes as are now starting to include a link to changelog News for practical AI. So if you do go to the show notes, you'll see that there's a link right there where you can get into the conversation very directly as well. I just wanted to mention that other, that other newer approach that we're, that we're starting to roll out.

Yeah, thanks. Well, as we kind of wrap the discussion of GPT2 up and before we share some learning resources, maybe it would be good to kind of summarize some, some takeaways from what OpenAI has done and from how the community has responded. I think one big takeaway that I have seen is that we can pretty much expect, as you've already alluded to, that OpenAI, Google, Microsoft and these other big players are no longer thinking that it's appropriate to kind of innocently publish all of their new AI research findings and the code associated with them. So to some degree I think we can expect that the days of just like everything going on GitHub all of the sudden and download all of the pre trained models I think is over to some degree, which is sad in certain respects and maybe appropriate as well.

Yeah, I think to tag onto that the age of any significant release automatically considering the issues around AI safety along with ethics, is part of the release at this point. And you know, and if you're, you know, for coming from more of a software development background that's, you know, it's been rare cases, very specific that you'd have to think that way because, you know, most software isn't inherently so powerful that it could be used for good or ill in many use cases the way some of these technologies are. So I think it's a maturing process that we're having here and I'm glad to see that OpenAI is the leading the way as they do and thinking about how to release responsibly, I still think the code's going to be out there and I think not only them, but I think now that with this you'll see a lot of other organizations researching this area since there's already been proven results. So I think it's upon us and we'll just have to, we need to roll into it cautiously.

Yeah, along with that, I think businesses are taking this seriously because, you know, it can affect their bottom line if there's ethical concerns that, you know, can actually harm their business based on the AI software that they're using internally. They're to some degree looking at this from a business perspective and seeing that there is some connection with these ethical concerns to both the perception of them and how it affects their, their bottom line. Along with that, I think, you know, of course a lot of people and you as well have already mentioned that there is a huge need to like yesterday we need to be researching methods to detect, you know, AI generated text. And I know there's certain efforts out there. I also realized I forget who I was talking to. It was at a conference and it's really hard problem. It's generating the text is a lot easier than detecting if it was AI generated or not.

You are so right about that. That AI safety conversation that it may very well be an upcoming episode. Hint, hint. It talks about that it's much harder to differentiate the real from the unreal than it is to simply create the unreal. It's a order of magnitude harder. So that's one reason why cautious release may be a good, mature way of doing it.

And you know, lastly, if you haven't noticed, AI for natural language is on fire everywhere. So it's like everybody's doing AI plus natural language and tons of great results. So I think one thing that you can look for as this year goes on is some pretty crazy stuff probably to come out of conferences like ACL&EM, NLP and Neurips around natural language and this sort of thing and along, you know, kind of the unsupervised or semi supervised sorts of methods. So definitely something to keep an eye on.

Yep, I agree with you. I'm really excited about seeing use cases for technology like what GPT2 is making available gradually here, combine it with what GANS can do. I think that's pretty fascinating. I think you talked about how businesses will be impacted, but I think that there will be a wave of new types of businesses being created with these new technologies as well. And I'm very eager to see what kinds of thoughtful things entrepreneurs come up with.

Yeah, speaking of that, in a couple weeks here we're going to be interviewing the CEO of Hugging Face. If you're following natural language and AI at all on Twitter and elsewhere, they are all over the place creating amazing things related to conversational AI. So I'm really excited about that interview. So stay tuned for that one. To close us out here, we always like to share some learning resources. If this conversation has sparked your interest in these topics and you want to dive in a little bit more, learn some of the details, maybe even try some of the methods. Of course we'll link to like the code and the repos and everything in the show notes. But we did want to kind of point you to a couple sets of blog articles that I think can really help you get started. The first of Those are on MLXplain.com there's one called An In Depth Tutorial to Allen NLP which Allen NLP is this package based around or a toolkit based around Pytorch. And they have implemented things like Elmo and Bert in the in the toolkit. So that blog post would be really good hands on start. There's also a kind of paper dissected article about Bert on the ML Explained blog. Then there's this other blog which I kind of came across recently and I wasn't aware of from Jay Alomar and he has a series of blog posts called, you know, the Illustrated Something. So he has the Illustrated Transformer which is talking about this transformer sort of model that all of these these releases are based around. And then there's an illustrated bert, Elmo & Co. Which talks about these encoders. I know I pointed you to these illustrated ones a little bit earlier. Did you get a chance to take a look at those?

Chris? I did. They're really good and I recommend thank you very much for pointing those out. I recommend to listeners that want to dive in, you know, these can be fairly complicated topics to ramp up on and the illustrated pages are really good for doing it. It may not be all you need, you may combine that with other resources, but it's another good one that you found.

There. Awesome. Well, this has been a great discussion, Chris. Thanks for thanks for all your insights and looking forward to talking to you again.

Soon. Sounds good. As you said, we got more interviews coming up and so have a very good week and we'll talk to you next.

Week. All right, thank you for tuning into this episode of Practically I. If you enjoyed this show, do us a favor. Go on itunes, give us a rating, go in your podcast app and favorite it if you are on Twitter or social network, share a link with a friend, whatever you got to do. Share the show with a friend if you enjoyed it and bandwidth for Changelog is provided by fastly. Learn more@fastly.com and we catch our errors before our users do here at Changelog because of Robot Bar. Check them out@rover.com changelog and we're hosted on Linode Cloud Servers. Head to linode.com changelog check them out. Support this show this episode is hosted by Daniel Whitenack and Chris Benson. Editing is done by Tim Smith. The music is by Breakmaster Cylinder and you can find more shows just like this@changelow.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next.