🤗 All things transformers with Hugging Face

One thing that we think is really interesting is that unlike generic model hubs like TensorFlow's hub or PyTorch's Hub, because our models are all of the same form, we can build a lot of tools and machinery around using them. So for instance, we have a visualizer that works for all of our models. You can just upload your own model and get really interesting visualization of its internal structure. Or this open source project called TextAttack built adversarial attack System and it's able to generically build attacks to any of our models in our hub. So because they all have the same interface, it allows people to do these really longitudinal research projects across everything that's going on in the hub itself. And then I should mention that now we have a kind of an inference API on any of the pages you can just type in some text and it will run against that model and you can even call that from your own code directly without ever running anything on your machine. Just run it on one of these servers and we even have a Twitter bot that we just put up last week where you can tweet at it and it will run a model against your Twitter. You.

Bandwidth for change log is provided by Fastly.

Learn more@fastly.com we move fast and fix things here at Change Law because of Rollbar. Check them out@roar.com and we're hosted on Linode Cloud Servers. Head to lino.com changelog.

This episode is brought to you by Digital Ocean Droplets Manage Kubernetes Manage Databases, Spaces Object Storage, Volume Block Storage, Advanced Networking Link Virt Virtual Private Clouds and Cloud Firewalls Developer Tooling with a robust API and CLI to make sure you can interact with.

Your infrastructure the way you want to.

DigitalOcean is designed for developers and built for businesses.

Join over150,000 businesses that develop, manage and scale their applications with DigitalOcean. Head to dot co changelog to get.

Started with a 100 credit again dot co changelog. Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changeog.com community and follow us on Twitter. We're Racticalai FM.

Well, welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International and I'm joined as always by my co host Chris Benson, who is a principal AI strategist at Lockheed Martin. How are you doing, Chris?

I'm hanging in there. How you doing, Daniel?

Doing pretty good. As we talked about the last couple of weeks, I've been ordering parts for a AI workstation computer and it's sitting next to me and it's running.

Oh, nice.

I am successfully, or at least it appears that I'm successfully overfitting a model on the gpu. So I'll have to deal with that, you know, after recording. But it's running and it's not overheating yet. It's kind of stable at a, I think, reasonable temperature, so I'm happy on that front.

So it's funny because, you know, as we're on this call, we're on Zoom, and in the video you have the data center in the background, so I just find it funny.

Yeah, a bunch of dgx, Nvidia machines or whatever.

Yeah, but that's not what you're using. The workstation.

Mine is much smaller, although it's bigger than I thought because after I put the GPU and the case would not close, so. Oh, I guess that just is like airflow.

Perfect. There you go. It solved itself. Yeah, I'm just doing okay. I did something really stupid this morning. I was reminded that I am a klutz. I fell when I was running and it looks like I broke a rib. And you'd think that I'd do something about that, but I'm lucky. I have a fourth year med student for a stepdaughter. So I called her up and we agreed because Covid is running rampant, we were not going to have me go to the emergency room. So she said the treatment would be the same either way. So I'm just kind of cranking through the day, doing my thing. And we're now we're recording. We're having fun.

Man, you're really pushing through the pain. For AI podcasts. There you go.

You gotta be practical.

Yeah, that's something.

Okay.

Yeah. Feel free to scream. Mute yourself and scream a couple times or whatever you need to do.

Okay, will do.

Well, going from that note to something completely different, as the show might say. We're really excited today because we have a follow up on a show that we did quite a while ago. Actually, this was episode 35, so quite a while ago we had plumbed along on from Hugging Face to talk about what they were doing. And now we're very excited to have Sasha Rush joining us, who is an associate professor at Cornell Tech and is also working at Hugging Face on a bunch of different things. And involved in the Transformers library. And so we're really excited to have you, Sasha, to hear more about Hugging Face.

Oh, thanks. Thanks for having me on.

Yeah. Before we jump into all of that, could you just give us a little bit of a sense of your background and how you came to into the field of AI and eventually into NLP and what you're doing now?

Sure, yeah. So I've been at Cornell Tech for the last year, and if you don't know about Cornell Tech, it's a new university. It's about seven years old, but we've had buildings for the last two years. Our buildings are right in the center of New York City on an island in the middle of the East River. So every day we kind of take a little gondola over to the island and teach courses there.

Ah, how romantic.

Yeah, it's a pretty fun place. And yes, I've been a professor here for the last year. Before that I was a professor at Harvard for about four and a half years. And before that I was a postdoc at Facebook, AI Research, also in New York. So that's my background these days. I have a lab here at Grinnell Tech and I work with the team at Hugging Face, who are in Brooklyn. So it's nice. Everything's kind of centered in New York City. Lots of interesting AI and machine learning going around here these days. So my background after graduating college, I worked as a software engineer for about three years. I'm kind of a person who very much enjoys coding, and I kind of have that as kind of the first part of my personality. I then went to graduate school to study natural language processing. When I got into natural language processing, I think I really got into it because I was very interested in language and particularly kind of the algorithms and data structures involved in studying and understanding how language works. At that time, I did a lot of machine learning, but machine learning wasn't kind of the primary way we studied language. There were all sorts of other aspects about kind of how computers and language interacted. And actually my dissertation was much more about, say, the optimization aspects of language in a discrete sense. Kind of how you construct trees that represent different linguistic phenomena and how these interact with kind of classical computer science algorithms. And when I graduated my PhD, I kind of graduated right into the beginning of really kind of intense deep learning for language and doing my postdoc at Facebook. Everyone was kind of intensely interested in how we could do translation, how we could do question answering kind of completely from data using deep learning based systems. So I kind of dived right into that world. I sat next to the folks who were working on Torch at the time. Then it was written in Lua and a couple years later they converted it to Python and it became Pytorch. So I've always been very fascinated by kind of the tools and structures that make it possible to do these sort of systems in a kind of open source way. Some other things I've worked on in the past, I worked on a library called Open nmt, which was an open source translation library written in PyTorch and TensorFlow. And we worked with a lot of translation companies, particularly in Europe, to try to build open source tools, to let them build their own kind of custom Google Translate like services. And that was a really fun project and it kind of tied together the research we were doing in my lab, which was on kind of questions of how to improve translation, how to speed it up, how to make it work on devices, with questions of how in an open source world these were used.

So I'm kind of curious since you kind of alluded to a little bit to one thing that's kind of happened in recent years in terms of how, I guess people maybe used to think about NLP and still do for many tasks as far as like computational linguists have been thinking about these things for a very long time, but now there's been all of this focus on kind of extending these tasks to maybe generalized machine learning type problems. Could you give your perspective on kind of how that shift has happened and like what that's meant both in terms of momentum in the field and people getting involved in the field and all of that. What are your thoughts on that?

Yeah, let's see. So I think there's a couple different perspectives. I don't want to make it seem like kind of data driven or machine learning systems were kind of new to nlp. There's a long history of use of learning both in nlp, but also kind of learning systems developed in NLP being used in other areas. So I think it's a field that's always kind of interacted with these methods in a kind of open dialogue. I think the phenomenon we're seeing now is kind of more extreme. And it's extreme for a couple reasons. I mean, one is the sheer growth of all these fields. We're seeing kind of exponential growth in conference sizes and paper submissions and kind of usage of this technology, which I honestly think is a great problem to have. But it obviously brings with it a lot of challenges. So they're kind of organizational questions, of kind of running communities or kind of trying to kind of make progress in this world. I think the other question is, what does it mean in terms of methods? And we're seeing lots of interesting things along those lines. I think that people in the field are adapting to the challenges that kind of come kind of from the world around. Like, as researchers, we're interested in solving the problems that exist now. And a lot of the problems in NLP are suddenly kind of data set problems. How do we construct interesting, novel and difficult data sets? How do we analyze models to understand what they're doing and how they're structured and what they're learning? Kind of societal questions of how do we understand what biases they might have or what issues they might bring or even how they might learn, like, from what. What signals are they picking up on? And so there's no shortage of interesting research going on. It's just that what's interesting is maybe less so, the kind of, how do you make the benchmark problem go up? X number of points?

So I'm kind of curious. I've been thinking about listening to this and, you know, we had Clem back on the show back. I think it was episode 35.

35, yeah.

Going way back.

It was before the Transformers library came out, which we'll definitely talk about later.

Yeah, totally. I think that was what I was thinking about was the fact that when we were talking to Clem, we were really kind of focused on, like, social AI and chatbots and similar tools and approaches. And then in that time between talking to you today and talking to Clem, you know, Transformers came out, and you guys really created the definitive Transformer library. And, you know, we've been talking about Hugging Face in the context of Transformers since then. And I guess how did Hugging Face make that transition? What caused that? And it's an interesting turn for the, you know, for. For the history of the company.

Yeah. So, I mean, I guess I should give some perspective. So I've actually only worked at Hugging Face for about eight months now. And honestly, I ended up working there because I was such a fan. I observed them in the same way that you did, which was as an external observer, seeing them make this transition so impressively from kind of working on chatbots to being this kind of open source powerhouse. And I guess as someone who. I guess, I mean, who knows what it means in open source, but as a competitor, as someone building his own libraries in this space, they were just doing it so much better than I was. And so I think that that always impressed me. Now, I should say, even before Transformers came out as an official library, I have memories of, well, I guess now we're getting into some of the technical terminology. When Bert came out as a paper, there was a kind of rush to port Bert to a Pytorch version. And I was working a little bit on this at my own pace. And Hugging Face very, very quickly put out their own version of this, maybe part of their chatbot library, maybe it was a separate thing. And I think it was really useful just to have that immediately right after the research came. And so I was really appreciative of that even at the time.

What's the state of Hugging Face now in terms of. I know that they raised a round of funding. It seems like from what I'm picking up on Twitter, that the team is growing a little bit. But from chatting with you before, it seems like it's still also very distributed. There might be some kind of creative relationships, like, of course you're in academia, but you're also with Hugging Face. So what's the state of the Hugging Face team now and how's it growing to support this really rich ecosystem of tools?

Yeah, so we have about 15 to 20 people, depending on how you count. We're mainly focused or entirely focused on these open source tool development. The main library is Transformers, which we've talked about, and kind of is this the center of what we're developing. But now there's also several other really interesting open source projects going on. So we have a project based on NLP data sets that now has almost 150 different open data sets that you can easily browse and download and use in a very efficient and kind of easy to extend way. We also have a library of tokenizers that's written in Rust, a low level library that lets you do very fast tokenization and training. And then all this is kind of joined together by a kind of hub of different models and structures that people have uploaded. And if you go to the website, you kind of see this kind of really rich ecosystem of different models, of different data sets and of different tokenizers that kind of build this all together practically. It is an interesting question of what the company is like. I mentioned earlier that I've been there longer now, probably in Covid than not in Covid.

Yeah, I guess that's true.

But it's always been a distributed company. There are. There's a team in Paris and a team in New York. It's about half and half, but now we Also have interns in California and some interns in China, some people in different places. So we mostly kind of communicate through Slack and through other distributed means.

Changelog News is the best way to keep up with the fast moving software world. We track, log and contextualize the coolest projects, the best practices and the biggest stories each and every week. Make changelow.com your daily destination or hit the snooze button and subscribe to our weekly newsletter that hits inboxes on Sunday mornings. Join more than 15,000 enthusiastic readers. It'll cost you exactly $0, and you can subscribe right now@changelog.com weekly.

So I guess, you know, we've alluded to Transformers several times now and kind of talked around it a little bit. For those who are new to the topic, could you kind of define what is a transformer? I mean, it's been a big, big deal in recent months and has really changed nlp, but a lot of people may not be familiar with it or have not kept up to date. Could you kind of just give us a basic run through from the way you see it?

Sure. So I think the term transformer really kind of implicitly applies two different innovations, and both of these were actually connected to each other, but both pretty transformative in their own right. I'll start with the first. So the first is the transformer as an architecture. So this is the particular kind of development of a very specific type of architecture that came out. And the kind of dominant architecture in natural language processing for about five years had been recurrent neural networks, particularly the LSTM network, and was used basically for everything that we did in the field. And the transformer proposed a different and in fact kind of simpler architecture that instead of kind of reliant on these recurrent connections, kind of connections over time, instead used a kind of random addressing style of architecture based on a mechanism called attention. And the way it works is that you basically have everything you've seen in the past ready to access at every point in time. And the main kind of neural network step that you take is the kind of soft random addressing over all your previous history, and you use that in order to compute the next stage in your sequence. So instead of kind of keeping a fixed length vector that gets transformed over time, you keep around everything and you basically search through it at every stage in the process. And this architecture wasn't kind of new on its own, right. But kind of demonstrating that it was more effective than recurrent neural networks, and that particularly could scale to both kind of Fast training and also very, very large models, better than recurrent neural networks was kind of a big breakthrough in the field. And the first results showed kind of large improvements on translation accuracy.

Just quick question. You mentioned attention and you sort of defined it in the larger thing, but just because it's a kind of a key aspect of that. Could you talk about what part of that was attention, just to differentiate it from the larger process?

Sure, yeah. And I should say the original Transformer paper has the title attention is all you need. That's kind of the key aspect of what makes a transformer. Attention itself is actually quite simple and it's actually kind of very kind of intuitively appealing idea. So imagine you have a set of objects, say five different objects, and you want to have a neural network. Decide which one of those objects you want to use. You might have a softmax layer where the softmax gives you a distribution, a probability distribution over which aspect you want to pick. So which of the five things you should choose? You could just end there, and if you ended there, we would just call it multiclass classification. What Attention does is it uses that distribution, the probability of each of the five things, and feeds that probability back into the model itself. So it would give a weight to each of the five items and then feed them back in with that weight. So imagine I have a sentence like the man walked the dog, and I want to predict the next word in that sentence. Those previous five words would be the five items I'd want to choose from. And Attention would say, how much weight should I give to each of those previous five words when trying to decide on the next word? So maybe I'll give it 80% to man, 5% to the etc. And use those in the next step of the process. All the transformer is, is a kind of repeated version of that game for say, six to 24 different rounds, where each time you look back at what you've previously decided, use it to feed it back into your network, and then use that to try to predict the next step along the line.

So you mentioned that this architecture also, in addition to kind of having this new structure, also allowed some performance benefits and scaling as well. I was wondering if you could just give a sense of. Because I know this is something people see out there, and in particular, I think there was a thread on Twitter about how many parameters are in the hugging face model hub and all of that. So I was wondering if you could just give us a sense of what are the sort of scale of models that are out There people hear about Bert and GPT2 and now of course we're getting flooded with GPT3 things. What are the sort of scale of these models both in terms of parameters and also the data needed to actually train them?

It's a good question. I never know these numbers offhand. Models range from 50 million parameters to tens of billions of parameters at the top end. In practice, some of the larger models, it's unclear how you would even use them, say standard GPU hardware. But scale has been a big kind of main aspect of kind of Transformers usage. But actually maybe let me take that question to talk a little bit about the second main innovation of the transformer. I talked about the architecture, but I think it's important to also get a sense of the second innovation. I think it actually matters even more. This is a kind of innovation that started around the use of a model called Elmo. There were a couple other variants, one called Cove, and then this all kind of peaked with the release of a model called bert. And the kind of idea behind these models is to take a neural network, in the case of bert, a transformer, and to train it on a very simple task at a very, very massive data scale. So in the case of bert, the task is similar to the one I described previously. You're given a bunch of words and you randomly remove some of the words and try to predict them back. It's a game that you can play yourself and try to get a sense of how easy or hard it is to do. Sometimes it's really easy, sometimes it's really challenging. But the point isn't the task itself. The point is to give the model a task that would require it to know something about language in order to complete and then train it at as big a scale as you can. So it's hard to give you a sense of this. I mean, one thing that's nice about language is you can store a ton of it in very little space. So if you have all of Wikipedia just basically fit it on your computer, and companies like Google basically have a non trivial amount of all the text that's ever been produced and, and so you can kind of take all that text, throw it into one of these models and then train it on this simple task. And it turns out that in the process of trying to complete this task, the model learns a lot about how language works. We say it learns very good features for language. So once you've done that, once you've kind of trained it on all the language that you have, you can then Apply it to a much smaller task that you maybe have a small amount of supervised data for. So this idea, which people call pre training is kind of central to how a lot of NLP works these days and also to how the Transformers library is designed.

So yeah, I think that's such a great and important point is that people kind of get hung up on the size of these models and it's kind of cool to talk about those things and in some cases annoying to work with them because they're so large and in some cases hard to perform inference with. But yeah, I guess what you're saying is, you know that the task that they're trained on is just intended to help them learn good features and then the task that you actually want to use them for involves some like fine tuning or transfer learning. Is that right?

Yeah, I think. I mean, I don't want to claim that this is finished as an idea.

Right.

I think a lot of the tasks we work on now will have a kind of fine tuning stage where you take the model and learn it for a given task. OpenAI has a slightly different model of what they're trying to achieve, which is they're not super interested in fine tuning. They want to kind of just use the model directly, kind of feed it some more sentences and try to directly kind of predict tasks.

Yeah, so there is this like, because I've seen and maybe you could kind of help us through some of this jargon. It seems like people talk about some of these models, they just like they have so much knowledge that you can perform a task that they just right off the bat, like I don't know if it's question answering or information retrieval or whatever it is without really much fine tuning. Is that what you're kind of getting at in that other model?

Well, I do want to distinguish kind of two aspects. I think that all the kind of state of the art models on kind of standard benchmark tasks all use some sort of fine tuning that's like become a very standard procedure and we kind of understand how that works. But to do fine tuning you still need some amount of supervised data. I guess we would say it's small to medium amounts, but you need something in domain for the task you're interested in. I think there's a lot of recent excitement for kind of a crazier idea, which is this kind of zero shot or one shot idea of just the model should know how to do your task immediately right off the bat.

I think that's where I was going because they throw around this idea of zero shot. And to some degree it seems sort of magical in many ways to people, I think.

Yeah. I don't want to say anything on record. It's on the research frontier.

Yeah.

It might turn out that that's the way to do lots of language tasks, but I think. Still an open question, I would say. So.

Turning to the transformer library itself, I'm kind of curious and recognizing the. That you've only been there at the company for a limited amount of time.

Yeah.

Do you have any insight into kind of the motivation that moved the company into this transformer library itself? Was it supporting the other operations or was it just something that was an opportunity that came up? What kind of took the company there as far as you're familiar?

That's a good question. The graph of the usage of this library on GitHub kind of blows me away. Like it went from no users to about 30,000. So I think they just hit on something that was like, I guess when you have a hit, maybe that changes the mode of thinking.

Yeah. So maybe you could describe like along with that, what is the sort of main usage pattern that people are kind of grabbing onto transformers for? I know that there are multiple, of course, like quite a few different things that you could use the library for, but what do you see as the sort of like the main thrust of what people are grabbing transformers for? What is that? And you know, how is that being supported? I guess.

Yeah, yeah. This is a great question. And I think in some ways you guys maybe have insight into this that I would be also interested to hear about. Let me start at the high level. One thing that fascinates me about kind of current usage of deep learning is that you have people who approach it from many different angles. And in one of our papers we kind of broke this down into three different classes. So we talk about there being architects, there being trainers, and then they're being end users. And I think within the ecosystem, Transformers kind of has different meanings to all three of those people. So if you're a company like OpenAI or like Allen AI kind of companies at the kind of cutting edge of research training, you use Transformers or kind of related libraries to try to build the next architecture or the next pre trained model. And that often means running these very large training jobs on multi GPUs over many days and then using transformers as a way to distribute your model through our hub and make it easy for people to use it or to adapt it for their tasks. If you're like an expert but maybe not kind of at the front end of the frontier of research. Another common use case is this kind of fine tuning use case where you have data for your company or for a given problem that you want to solve and you bring that data into the library, use it in training mode to fine tune on your data set. It may take a couple hours and require some GPUs, but out of that you get a really accurate model for the task you're interested in. But then at the other end you have just end users who want to use the library as a way of kind of performing kind of standard NLP tasks. You might want to use it as a way to do summarization or translation or named entity recognition or question answering. And you can often just use it completely in inference mode, maybe not even using Python, just kind of taking a pre trained model, using it directly for your task in that kind of setting. So I think all of these people are within the machine learning ecosystem, but they kind of have different end goals or different use cases. And I think we're trying to aim to support any of those outcomes.

So I know you have a model hub and was wondering if you could talk about what users can find there and start incorporating into their own projects. What does the growth of that hub look like? Just what kind of ecosystem has developed around it?

Yeah, so the model hub is kind of part of the open source library. If you want to use a model in the library, you say model load and you pull off, you just pull it directly down from the model hub and you can do that with any of the models that are there. We have kind of a set of models that kind of have brand names that are very often used. So those include models like GPT2 or variants of BERT or Roberta, or new models like this model called Bart, or a model called T5. But then it also includes a long tail of other models from the community. So this includes models that are pre trained to target, say biomedical text or extraction from scientific documents, or models that are trained in many different languages, kind of by the communities interested in those languages themselves, or models that are experimental or try to do other things or one popular aspect is models that are very small models that you could run on your phone. So the idea of the model hub is to kind of have all of those have the same API and have the same easy way to use them. And one thing that we think is really interesting is that unlike kind of generic model hubs like TensorFlow's hub or PyTorch's Hub, because our models are all of the same form, we can build a lot of tools and machinery around using them. So for instance, we have a visualizer that works for all of our models. You can just upload your own model and get really interesting visualization of its internal structure. Or this open source project called, I think it's called Text Attack Built Adversarial Attack System and it's able to kind of generically build attacks to any of our models in our hub. So because they all have the same interface, it allows people to do these really kind of longitudinal research projects across everything that's going on in the hub. It's. And then I should mention that now we have a kind of an inference API on any of the pages. You can just type in some text and it will run against that model and you can even call that from your own code directly without ever running anything on your machine. Just run it on one of these servers. And we even have a Twitter bot that we just put up last week where you can kind of tweet at it and it will run a model against your tweet.

Yeah, that's great. I was wondering before, before we leave the topic of the open source projects, you also mentioned these other libraries, tokenizers and nlp, which includes the data sets and evaluation metrics. How do those fit into the puzzle and maybe interact and influence one another?

Yeah, I mean, end of the day our interest is in building open source nlp and I think there will continue to be kind of new variants of transformers and new pre trained models. But kind of, as I mentioned earlier, an increasing area of innovation in NLP is to try to find the right data sets to kind of challenge these models in interesting ways. And so there's a lot of energy in dataset construction these days and a proliferation of really interesting data sets of different sizes and scopes. And so Tom Wolf, who's our main open source engineer, got very passionate about building up open source data sets and build a library that makes it very easy to use these models in Python and really makes it extremely efficient to use kind of complex data sets directly within your code across kind of many different aspects of nlp. And so you can, we have a website that you can go to where you can kind of browse through any of these data sets and kind of use them in various tasks. And one nice aspect of this is that we have a lot of examples of how to use transformers and they had a lot of kind of custom data set code just to run the examples. But now that Code has all kind of been factored out. You can just kind of pull it in from NLP and then run the examples, kind of focusing on the machine learning parts.

Foreign.

We deserve a better Internet and the Brave team has the recipe for bringing it to us. Start with Google Chrome. Keep the extensions, the dev tools and the rendering engine that make Chrome great. Rip out the Google bits, we don't need them. Mix in ad and tracker blocking by default, quick access to the Tor network for true private browsing and an opt in reward system so you can get paid to view privacy respecting ads. Then turn around and use those rewards to support your favorite web creators like us. Download Brave today using the link in the show notes and give tipping a try on changelog.com.

So to take the conversation in a slightly different direction for a moment. I know from talking before the show that you put together ICLR and you kind of managed that process this morning and for which is a research conference and I'm really interested at this point. You know, we are in the time of COVID 19 and so much has changed across all of work, but particularly conferences. Many of them are going online, becoming virtual like that. Really interested in what that was like and what, you know, what your experience doing it this way was and, and what worked, what didn't. That kind of. Again, just curious because I think a lot of people are kind of waiting to see what conferences are turning into and do they want to continue to go down that route or something.

Yeah. This year I was the general chair of iclr, the International Conference of Learning Representations. It's a big machine learning conference and really the only one focused completely on deep learning. And it was interesting. I had the chance of being the program chair for the conference last year where we had the conference in New Orleans. And then this year I was a general chair and by about December we were getting prepped. And then by February, March, it became increasingly clear that we weren't going to be able to have this conference live. And so I think we were the first AI conference to really have to be completely virtual. We had about a month and a half before the conference to really come up with something new. And we had this wonderful team led by the program chair this year, Shakir Mohammad, and we wanted to do something that kind of fit the spirit of the conference. And so we sat down and wrote a website for the conference from scratch and we built a website that was based around this idea that everyone in the conference would be in kind of a zoom like, sorry, a slack like chat room we used an open source platform for that and that every paper would have its own page with a video of the work and a chat room for that paper so people would be able to kind of talk about it or discuss it within that setting itself. And in addition, we build out kind of a bunch of social gatherings that people could have and a kind of calendar for the whole event. And the kind of main challenge is how do you run a conference asynchronous in this way? We didn't really think it was possible to have everyone in the same place at the same time. And so we wanted it to kind of use things like chat rooms that kind of feel more asynchronous, particularly with an international audience. And the conference itself actually was really fun. We had actually a pretty large increase in attendance over past years. We had people from all over the world, particularly from some places, that would have been difficult to attend a conference in other years. And a ton of engagement. A lot of the posters were viewed a tremendous amount of times and maybe about 100,000 messages over the chat system over a couple days. I think there were challenges. I think it's hard to get the same kind of spirit of having coffee or kind of just chatting informally in this sort of event. Things like Twitter are helpful, but don't have the same kind of intimacy. But there are also kind of nice things. We ran these kind of mentorship sessions where one person was able to chat with 10 to 20 folks who were interested in mentorship. And a kind of one to many model that actually I think might have been difficult at a conference, but kind of works actually pretty nicely over zoom. Anyway, it was an experimental setup. Since then we open sourced all the tools that we built for the conference. You can get it online if you search for miniconf. And the software has been used for about five or six other major conferences since then, including ACL this year, which is the big NLP conference, and ICML which is another machine learning conference venue. I don't think it's something we've cracked it, but in the meantime, it's nice to have something we build as a community.

Yeah, I, I attended the, the conference iclear and I was super impressed with everything that was put together, especially given the timeframe. I know you, you must have had some very late nights fueled by very much coffee. So congratulations on, on, in such a short time period putting together something that was so good. I know one of the things that I appreciated, you know, I've been to other research conferences in person and you Know, posters or talks or something like that. There's just so much going on that it is hard to kind of do that. Like you can't go to this talk at the same time as this talk and it's hard to find that person afterwards and ask them some questions about their work. Maybe you walk by their poster or something. So it was kind of nice to just scroll through and look at the different videos, especially given the time zone differences, and shoot the authors a message that they could respond to asynchronously so that that question didn't get lost or something like that. I found that extremely useful. What are your thoughts on assuming maybe that at some point in the future research conferences will have an in person component again? Do you see a sort of hybrid scenario developing? Because I know one of the things that like with Neurips and all of that was a struggle for so many years were people getting visas as well, which was just such a shame as like so many people from Africa or from Asia that were doing amazing work but couldn't actually be at the conference because of visa issues or cost issues or whatever it is. So how do you see that future happening?

Yeah, it's a question we're talking actually a lot about at ICLR right now. I don't think we have an answer and I think a lot of it will depend on kind of what the world looks like in a couple years. So one thing we're committed to at ICLR is having the conference at venues in other locations or kind of locations that have kind of not been visited as much in the past. So one thing that was very disappointing was that this year's conference for ICLR was supposed to be in Ethiopia, in Addis Ababa. And we were all really disappointed that we couldn't make it out there. It would have been a really interesting event. So hopefully we will continue to kind of have conferences in kind of a wider range of locations. But I think as I was saying earlier, all these areas are experiencing such kind of hyper growth that kind of ways of, kind of dealing with scale that doesn't lose a kind of sense of interaction is a kind of major challenge for the community. So I think we need to kind of be creative about ways to handle that problem and ways of kind of maybe giving people the same experience that I think, at least I feel like I had when I was a first a graduate student that kind of inspired me to continue in the field. So I don't know what that'll look like. Maybe it looks like something more distributed with a virtual component.

So kind of wondering, also turning the corner a little bit on just NLP in general.

Yeah.

And you know, you're doing the work that you're doing. You're right at the center of the NLP world in that way. And it certainly, you know, Daniel and I talk all the time on these episodes about the fact that the last couple of years has felt like, you know, NLP has really come of age. You know, you might say golden age of NLP is how it feels like we're in and kind of before that, you know, we had seen like CNN's have their moment and stuff as we've arrived where we are so far in nlp. You know, what does the future look like to you? What kind of big challenges are open and should be focused on, you know, what are your thoughts there on from this point forward?

That's a hard question.

Big one.

Yeah.

In some ways, as someone who's been working at NLP for a while, it's been really neat. I mean, I think it's way better than I could have possibly expected. Seeing things like translation get to the point where it's at now is just awe inspiring to me. Like, it's such a useful thing and have it work the way it does is awesome. So what are the challenges now? I think there's a bunch. I think computer vision, for all its successes, has also had a lot of issues. And there's a lot of conversation in NLP about kind of how to avoid some of the issues or to kind of have those conversations earlier rather than later. Things like what we've seen with facial recognition as a technology and kind of questions about efficacy. There is, I think, a kind of challenging point. And we've somehow managed to solve a lot of the natural language processing questions without solving some of the computational linguistics questions, like things work, but we have no real sense of why. And as a scientist, that can be a little bit frustrating. Like we don't really know what signals these models are using to make predictions. And it's very hard to know or to even ask that sort of question in a falsifiable way. Why did this model classify this sentence in this way? Why did it decide to choose this decision? I mean, these models are, at least from a probabilistic sense, completely global. And so it's kind of challenging to kind of do any sort of analysis along those lines. But then more kind of practically, I think there's a lot of practical questions that are not solved yet. You mentioned this idea of dealing with massive, massive models. It's not clear if we're going to need hardware that is 100 times bigger to run these models, or whether you can use pruning and distillation to make them super small, or what does it mean to run it locally or does it just make us more reliant on cloud systems? I think these all become interesting systems research questions in the short term.

Awesome. Well, we appreciate you taking a stab at the future predictions because I know, I think we said on the podcast before any of the predictions that we make I feel like are definitely going to be false because it's always something unexpected that happens. But I appreciate you giving, giving your perspective, being part of the kind of the center of all of this work, and appreciate you taking time to talk with us and kind of explain a bit about the Transformers library and things that are going on in nlp. Thank you so much for your contribution to the community as well in terms of helping, you know, conferences and really pushing forward open source. So appreciate you taking time to join us and looking forward to digging into all the great things that Hugging Face is releasing and is doing.

Thanks so much. Thanks for having me on.

If you're listening to this in the month of July, you've got a shot at some free goodies. We are doing a giveaway in celebration of our friend and open Source Whiz Zeno Rocha's new book, 14 Habits of Highly Productive Developers. If you don't know Zeno by name, you may have heard of his wildly popular Dracula theme. It's an awesome dark mode theme for text editors, terminals, etc. And we have three bundles of Dracula Pro and 14 Habits to give away for absolutely free. That's a sixty dollar value and there are three ways to enter. You can be the reviewer, the socializer and the recommender. Hit up the link in your show notes to get started. There will be three lucky winners and you could be one of them. Thanks to our longtime sponsors Fastly, Linode and Rollbar for their continued support Do Breakmaster Cylinder for our amazing Beats and to you for listening to Practical AI. We appreciate your time and attention. That's all for this week. We'll talk to you next time.