Green AI 🌲

We're researchers, we mostly communicate with the research community. But I mean there's stuff to be done everywhere. Thinking about, you know, thinking about efficiency. You don't have to persuade anybody that if all the other things being equal, if your tool runs twice as fast or takes half the amount of memory, everybody wins.

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com Our feature flags are powered by LaunchDarkly. Check them out@launchdarkly.com and we're hosted on Lino Cloud servers. Get a hundred dollars in Hosting credit@lino. We deserve a better Internet and the Brave team has the recipe for bringing it to us. Start with Google Chrome. Keep the extensions, the dev tools and the rendering engine that make Chrome great. Rip out the Google bits, we don't need them. Mix in ad and tracker blocking by default, quick access to the Tor network for true private browsing and an opt in reward system so you can get paid to view privacy respecting ads. Then turn around and use those rewards to support your favorite web creators like us. Download Brave today using the link in the show notes and give tipping a try on changelog.com. Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@change dog.com community and follow us on Twitter here. PracticalAIFM.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist at SIL International and I'm joined as always by my co host Chris Benson, who is a principal Emerging Technology strategist at Lockheed Martin. How you doing Chris?

I am doing very well.

How's it going Daniel?

It's going great. It's warmer now in the us A lot of people have been having some issues particularly down in Texas and other areas. So this is for those listening later in the podcast. This is February of 2021. A lot of snow and cold weather in the US here.

So a couple of people on our team at work are in Texas and we've been getting all the, all the stories when they're able to connect and stuff. So I think, yeah, I think they're, they're getting through it finally. Thank goodness. It was pretty, pretty horrible. But yeah, in the meantime I am enjoying my 70 something, you know, my 70 degree plus weather outside, spring like and I'm kind of sticking my tongue out on them on Zoom meetings.

Yeah, it's always interesting during these particular types of events because you kind of just assume that people have all this like redundant, fault tolerant, like infrastructure going on for like their APIs and other things. And these sorts of events really reveal like that is not the case. Like I know like one of the APIs we, we frequently use, like apparently on an on prem server in Dallas and they did not have power. And you know, you learn new and interesting things like that.

You know what? After the past year, there's nothing that surprises me anymore. Not now. Global pandemic, all sorts of strife, you name it. I mean, I'm just. Yeah, nothing phases me now.

Yeah, I'm glad to hear you've built a lot of robustness into your personal life there, Chris.

There we go. I laugh a lot, I snicker a lot. That's how I cope.

Yeah. Well, a few months ago actually, I think it was one of the researchers at SIL that I work with called Gary Simons. He's been a linguist and programmer, computational linguist, translator type researcher for decades. And he sent me this link in our Skype communications said, hey, this is a really cool article. You should think about having this on your podcast. And there's an article called Green AI from Communications of the acm. And I'm really happy today because we get to materialize what what Gary saw and what he recommended to me. And we've got Roy Schwartz and Jesse Dodge with us. Roy is a senior lecturer at Hebrew University of Jerusalem and Jesse Dodge is a postdoc at the Allen Institute for AI. And they were both authors on that article. Welcome guys.

Thanks for having us.

Thank you.

Yeah, if both of you could just give us a little bit of a background about yourselves, that'd be great. Why don't we start with Jesse?

Sure. So I finished my PhD from Carnegie Mellon in the Language Technologies Institute last year in 2020 in the pandemic. Although I spent most of my PhD at the University of Washington in Seattle. And part of that time I spent working at the Allen Institute for AI, where after I graduated, now I'm back as a postdoc full time. So we wrote this article, I think back in, we were thinking about this for quite a while and then wrote this back in 2019 and really got it out in 2020. Now here, even though the offices are closed, I'm still here in Seattle and I am on the Allen NLP team once again.

Awesome. And what are you specifically working on?

So my research sort of falls under two broad umbrellas. The first is related to efficiency. Similar to this Green AI idea that we'll get into. I work on making models more efficient along the number of dimensions that they have in terms of the complexity in terms of inference, generally related to any way that you can measure, like the total computational cost of getting some kind of experimental result. And then the second pillar of my research relates to reproducibility, where I created the Natural Language Processing Reproducibility Checklist that was used at, I think, four major NLP conferences now. And I've published some work on how we can make the science of machine learning and natural language processing more reproducible.

Yeah, that's awesome. Well, you're working on two things that are just desperately needed in terms of focus. So, yeah, I commend you in terms of that. And yeah, it's really great to hear. Roy, what about yourself?

Hi, so I'm Roy Schwartz. I'm a senior lecturer, which is an equivalent to assistant professor at the Hebrew University of Jerusalem. I'm currently in Jerusalem. I joined the Hebrew University last summer, and before that I spent four years in Seattle, where I got to meet Jesse, fortunately. And I was a postdoc and then a research scientist at the University of Washington and the Allen Institute for AI. And these were four wonderful years. But now I'm back home, similar to Jesse to some extent. I also came from the university where I did my PhD and so I took a break and came back. My research also spans two or maybe three dimensions. One of them is similar to Jesse, efficiency and trying to think about ways to reduce the cost of AI and NLP in particular. And the other is trying to get better understanding of this technology now that we have models that are becoming so big and so good at what they're doing. But at the same time, it's very hard to know why they're doing certain things, why some things work and some don't. Why do models reach certain decisions? I'm particularly interested in the role of data in all of this. How do our data sets look? What do they contain? What kind of phenomena are encoded in them? And I like to make connections between all of these goals, between understanding and. Between understanding our data and between making things more efficient. And these are some of the things that I'm most excited about.

Awesome. Before we move on, what is your general impression about sort of progress in this process of trying to make our models more interpretable and understand more about them? Obviously you're doing work in the field, so hopefully you see progress in that. But as an industry as a whole, where do you think we are on that journey.

That's a great question. So, as you said on the one end, we're making tons of progress. I mean, lots of very smart people are working towards developing method to probe models, to kind of, kind of poke them and ask them, do you know syntax? Do you know world knowledge? Do you know this? Do you know that? And we're developing methods that are more and more sophisticated to get this information. At the same time, the core questions that I think will make a huge impact if we're able to solve them. And I'm not sure if these questions are even solvable to some extent, and I'm happy to talk about, even though it's not the topic of today's talk, is how do we get models to explain what they're doing, to explain it in a reliable way, in a way that's. I mean, I'll just say one thing. I mean, when you ask a person why did they do something like this that they did, the explanations are often also not. I mean, they might be post rationale of things that it's hard even for us to say why we're doing certain things. And we're conscious creatures, so machines. It's much harder to get this. But we're trying to.

I appreciate that as we look at this, we're talking. I'm looking at your green AI article here again, and I'm just kind of curious, what was your motivation for putting this out? And probably I should ask as part of that, what is green AI initially? How did you decide that this was the thing that you needed to get out there to the world? Because this is a topic that often gets left out of AI ethics and such. Having worked in that field for a while, we can go back to that in a little bit. I'm curious what your motivation was, though.

Yeah, so I think part of it was some conversations that Roy and I had. Again, this was back in 2019 when we were both at the Allen Institute for AI and we noticed that there was this increasing trend of larger and larger computational budgets used for some of the research papers that were published in nlp, we looked around and found not only did we notice this, but there were a couple other pieces of work that had also noticed this trend. So back when I started my PhD back in 2013, I could run my experiments, often on a used laptop that I had purchased off of Amazon. And it was kind of slow, but I could run most of my. I could train my models in a few minutes or an hour maybe, and it worked. And that was okay. And then we noticed in 2019, we were like, wow, a lot of these models don't even fit on a single gpu. And we have to rent cloud instances to be able to actually use some of these models. Plus, in some cases, papers would do, for example, a tremendous amount of hyperparameter optimization, or they would train on a huge amount of data well beyond what we could do even at a good institution like the University of Washington or AI2. And one interesting thing, and this has really been followed up by some concrete research, is that we do find significant improvements in performance across a lot of tasks just by scaling up these models. So language modeling, for example, has been a pretty foundational task in nlp. What we found is that training models to do well at this task of language modeling, if you train a large enough model on enough language data, then that model can do some other tasks that we're interested in as well. So it somehow learns some kind of representation of language that's useful across a wide variety of tasks. But to get there, we saw just huge computational budgets used for a number of these papers. And interestingly, I mean, we wrote this a while ago, but the trend has not slowed down. So this is something like. Roy and I are still working on similar motivated pieces about how this is really driving a lot of research in our field. These massive scaling laws, for example, are pushing state of the art and also getting a lot of attention. And our field is interesting. You can view our field through that lens now and see some interesting results.

Yeah. So I'm curious. I have my own thoughts about how I might answer this question, but I also haven't done the amount of thinking that both of you have. So I don't know, maybe Roy, if you want to comment on this or kick it back to Jesse. So that trend has been continuing and like we're seeing those sort of improved results in some areas along that trend, like in language modeling. So why is that a problem? Or like what sorts of problems or red flags does that bring up? I guess, yeah.

I mean, I think it's interesting because Jesse and I bring complementary motivations for tackling this problem. So, I mean, when I started thinking about these things, I mean, yes, I was having discussions with Jesse about this, but I'm a person that cares about the environment and I try to make personal choices, that I ride my bike to work because it's healthy, but also because it allows me to not drive my car. And I try to turn the light off when I leave the room, do these simple things that don't you Know, don't matter much at the global scale. But, you know, I make my personal choices and then I go to my office and I. I don't know if you've ever seen a gpu, but this is a very loud machine, very. A machine that emits a lot of heat. Yeah, yeah. And kind of we're running stuff, you know, like, okay, let's just push a button and Suddenly, you know, 5 degrees or 10 degrees up in your room maybe, but not in your planet, hopefully. And kind of. It's been something I've been thinking about quite a bit. I mean, what's the total impact of our field? And Jesse and I have been talking about this and then I think in mid-2019 or early mid-2019, a paper came out from the University of Massachusetts, led by Emma Struble and her colleagues, that tried to quantify the CO2 impact of large scale NLP experiments. And she came to the. She and her colleagues came to the conclusion that one of the most expensive experiments that run the trainer model in a process called neural architecture search, which basically means we're going to train a bunch of models and select the best one. But when I say a bunch, I'm talking about thousands or tens of thousands of experiments. And she computed, using some rough estimations to be said, that the amount of CO2 emitted by this process is equivalent to the amount of the life term omission of five cars or several flights or. I don't remember the full detail, but I mean, something that. I think it was five car.

I remember this coming out and I was also shocked.

Yeah, Daniel and I actually talked about this in an episode way back when that came out. I remember us just commenting on it.

Yeah, everybody was talking about it and really hit me in a place that I. This is something that I was thinking about and I was really happy to. I mean, I was sad to see that, that my intuitions were right in some sense. I was kind of hoping that maybe it's not that bad. And then Jesse and I were having discussions along with other people at AI2 and kind of we were saying that this is something we need to do something about or to make the community more aware of it. And we were thinking about, I mean, AI2 is an institution that our goal is. I mean, I'm no longer working there, but at the time I was working there to do AI for the common good. And this feels like a natural fit for the goals of their organizations. And we got Oren Etiony, the CEO, and Noah Smith, who was My manager and Jesse's advisor at the time on board. And we wrote this piece and just hoping to get people thinking about this, not necessarily thinking about this, in terms of finding more accurate ways to quantify how much energy is omitted and how much are the cost of these experiments and trying to encourage the community to work on more efficient solutions that would allow us to reduce these costs.

Yeah, one way that Roy and I think one thing that Roy just mentioned is that we brought different perspectives to this. I completely agree with everything that Roy just said. That's super motivational. I think that's very important going forward, that we keep track of CO2 estimates and we do a great job at that. There's another side to this also, which we write about in our Green AI paper, where we talk about the sort of research inequality or inequality in the research community, where some of these experiments really could only be done by sort of the 1% of the research community, those that have access to tremendous numbers of GPUs or just lots of machines. So one question that we address in our paper is, is this valuable research that we should treat on the same level as like, other types of research that can be done primarily motivated by just a good idea rather than really expensive experiments? And so both of these are sort of negative consequences of this increasing trend that we observed. And one interesting thing, I think this is an interesting thing, sort of back in 2019, going back to that Struble et al paper, I found that through a number of conversations that I had had and also just the general information I saw online, before Emma and her colleagues wrote that paper estimating the CO2 emissions, there was an understanding of how some work was very expensive, how some work was, quote, boiling the ocean, for example, just to get a 1% improvement or half a percent improvement on some task. And so when Emma wrote that paper, I was surprised, but again, I felt similarly to Roy. I was surprised. I wish I hadn't been surprised by the results that I saw. I wish they had claimed that people were emitting less CO2, but it really did capture her paper and then our paper as well. I think these got so much traction partly because we were outlining a trend that other people had also noticed. And yeah, like I said, that trend really does. I think we focus on two facets. There are probably others, but the CO2 emissions and also sort of this research inequality are both direct consequences of that increasing trend.

Hey, friends. This episode of Practical AI is brought to you by Kodish, a podcast from the team at Heroku. That explores code, technology, tools, tips and developer life. There's tons of great conversations on the Codish podcast, so I would encourage you to check it out and subscribe. But in particular, I wanted to bring to your attention two episodes, episode 98 and 99, where Julian Duquet explores the ethical and technical sides of deep fakes. The rise of manipulated pictures and videos and other forms of computer generated media are able to cause uncertainty and doubt in what we see and hear online. So how are we able to use these tools for good, if at all? Here's a sneak peek.

Let's say we want to do a.

Deep fake of my voice and we.

Train the model and we have enough data and everything.

This will be also able to imitate.

My accent, for example, like how I pronounce English and the strong pieces of my accent or is not there yet.

It really depends. If there would be a person with similar accent on the input, then it would be fine, but it's kind of cheating. You can think it's cheating because we're reusing accent of a different person that's similar to your accent. But if it would be like an American native speaker or a British, a person with a British accent or like whatever, whatever other accent, then it will kind of be a mixture on the output. So we're not there yet in terms of converting accents. It's a little bit more difficult than we initially anticipated because like when we started the company we thought it was, you know, we'll kind of solve it in a year or something. But then it turned out that oh no, we're here for, we're here for much longer.

Check these episodes out. Links are in the show notes to both episodes or head to Heroku.com podcasts to listen and subscribe. Again, check the shows for links or go to heroku.com podcasts.

So you brought up something that really kind of got my brain really going there for a minute. And I was, it was thinking about the fact that, you know, this really can matter a lot, even if not a lot of practitioner. You know, the number of practitioners in AI relative to all the people producing CO2 is quite small. But you mentioned going through all these models and when we're doing things like hyperparameter optimization and trying, you know, little adjustments to architectures all the way through. And then one practitioner doing work is essentially, you know, being thousands of practitioners on a per model basis as they're trying to hone in on that, it really amplifies the impact of what can happen. I mean, so, so I Guess you know, it's not. It's less of a problem that a very few people are doing and more of a problem that. Because that amplification is quite outsized relative to the number of people doing it. Am I getting that right? Am I understanding the problem and the way that you're thinking about it or am I missing something there?

So I'm not 100% sure that I understood you. So let me try to say where I think this is going. I'm assuming you're talking about the environmental.

Yeah, environmental impact.

Because the inequality aspect I think is pretty clear that I mean a very small proportion of the community can afford to run these experiments and kind of when we're thinking about the environmental effect, then some people argue, and I'm not sure I disagree even that it's not so bad because these experiments are being run just a handful of time and I might agree on that. I must say there are different ways in which the AI community is contributing so and so, quote, unquote to the emission of CO2 to the atmosphere. And probably the one that's easiest to measure is the most expensive experiment. That's perhaps one dimension you can also think about. I mean the entire, the amount of training being done by the entire community. And probably most influential in this sense is the cost of inference, of cost of taking a model that's been trained and running it. And this is one operation is very cheap, especially obviously compared to training a model. But this is something that happens at scale. If you think about, I don't know, the amount of Google search queries that are being run per day or the translation or the number of videos being edited or recommendations in various websites. So there's different dimensions to these problems. And I think what we're trying to promote is not so necessarily to say, look, we're boiling the ocean, as Jesse said, quote, unquote. But I mean, we don't know exactly what is it that we're doing. And let's be more honest about it, let's do a better job at reporting and let's try to reduce these costs. It's hard to argue against. I mean, who doesn't want cheaper models, right? It's obviously that other things are if cheaper models perform slightly worse and maybe this slightly worse translates to slightly less revenue, then maybe cheaper is. There are different ways to define cheap. So I think what we're trying to promote is to get more people thinking about it and not just improving another epsilon on the accuracy level.

Yeah, that's super helpful. I think you know, one of the things that's running through my mind is, I guess, like talking about, you know, what are the other options, what does it mean to do green AI? And I have this parallel in my mind. So I come from a physics background and like, if you're in high energy physics now, like, there's just been a progression of larger and larger particle accelerators, right? And now if you want to do high energy physics, you're going to spend some time at CERN in Switzerland or whatever, just because no one has another cern. Right. Like, they're, they're just not there. So, like, is there another option for. I'm thinking particularly Jesse, of what you were highlighting in terms of the research inequality. I think that's a really, really great point. Like, what can we do in terms of reducing that inequality? And is there something more that we can say other than like, tough luck, go work at Google or somewhere that has these, like, amazing resources, you know, seemingly endless resources to do these massive experiments?

Yeah, so that's a great question. I think this is something that comes up a lot, is sort of the relationship when we talk about green AI. Sometimes somebody will say to us, oh, but in biology it costs so much to do any experiment because you need a wet lab and because you need some equipment, and you just can't do it without that equipment. So is it bad that some experiments in our field are expensive? And I think the answer here is really that in the computational sciences and in machine learning and NLP in particular, there are a few things that we can do that make future comparisons against our work with smaller budgets easier. So one example of that might be, sure, I train a model on all of the language data on the entire Internet, but I can also evaluate that same model after training on only a fraction of that data. And if I do this, let's say I train and evaluate. Evaluation in this case is typically pretty inexpensive. So your evaluation set, your data set that you evaluate on is often much smaller than, it's like, you know, a tenth or even smaller of your training size. So one thing that we can do is just checkpoint our model or evaluate it regularly throughout training, and then a future researcher will be able to come up with a new idea. Let's say they have a new model that they want to evaluate and they can compare against some of those sort of smaller budget evaluations. So for us, the point here is that in our field we really do have a few ways that we can sort of build in these sort of low budget comparison opportunities. And that enables not just future comparisons, but that really drives the sort of competitive nature of our field where instead of trying to improve just the absolute best found performance, somebody could try to find a better performance efficiency trade off where at a low budget, their new idea, a low budget for the number of parameters in your model or the total number of experiments of hyperparameter tuning, or the amount of training data you use along any of those dimensions, somebody else might come along and try to compare against your work specifically in those sort of low budget regimes. And so I think here that's a key difference between our field and physics, like you mentioned, or we often hear biology. And really if you think about it, if you're training a model and it costs you say a million dollars to train on all of the Internet, spending an extra $10,000 on just evaluating that model, spending an extra 10th of 1% or some small fraction of your total budget so that other people in the future can, they can have an opportunity, they've got that hook to compare against. That is one way that we can help drive the overall cost down by promoting that kind of competition.

Yeah, I totally agree with what Jesse said. I think presenting another angle of this. So currently there are certain norms in our community and there are certain ways of, I mean topics of research that gets more visibility and more credit from the community while others aren't. And I don't want to say the naive assumption is go work at Google as you said. But the fact is that when we were thinking about this paper a couple of years back, we were doing a short survey of papers in acl, that's the top venue for our field and in other similar venues in other fields of AI. And we had a very hard time finding papers that focused on efficiency. Most of the papers we were looking at were trying to say, okay, we did this and this and that and we got some better improvement here and this and this and that and we got some tenth of a percent better on some accuracy or answering questions tenth of a percent better or translating a fraction of a percent better there. And what we're trying to argue that this is not a good balance. We want to see it's good that people are working to make our models more accurate. We're not arguing that this is not important and similarly we're not arguing that the big models aren't important. They're making huge contributions to our field. But we think that a larger chunk of the research efforts should go towards trying to find solutions that are not epsilon better but are twice as fast. Or take 10% of the memory or what have you. And we're trying to work with the research community by providing ways to publish this work. For instance, we've established tracks. And tracks are kind of like, you can think of it as topics in major conferences where when we were working on some of our work that tried to promote efficiency or presented an efficient solution, as I said, that works five times faster but doesn't improve the performance, we had a hard time deciding where to send this paper to and where we would get the the best audience to appreciate it. And what we were able to do in the past year is to set up a green NLP track or an efficient NLP track in our conferences that allow works that focus on that to get published and to get the visibility that they deserve.

Yeah, that's great. And I think another thing to build on what Ro just said is our community. I think one strength of the research community is really that it's just a collection of individuals all trying to do the best work that they can. There is no overall governing body. So when we think about how can we get our community to focus on more efficient approaches, it's kind of tricky. It's just not possible for us to say some fraction of the work should cover this topic. So instead we thought a lot about the types of incentive structures that impact people in our field. And creating this track, as Roy just mentioned, is one of the ways that we can promote this and provide an opportunity, sort of lowering barriers for publishing work on work that promotes efficiency.

Have you heard about Knowable? It is an awesome new platform for learning from the world's best minds anytime, anywhere, at your own pace, through audio. Learn about the performance benefits of a plant based lifestyle from NBA All Star Chris Paul. Or how to launch a startup from Reddit co founder Alexis Ohanian. There's even a 10 lesson course from astronaut Scott Kelly. Here's a sneak peek.

We learned a lot up there. But what can you learn from a life in space? The answers might surprise you. In this knowable course, I want to share some of the things I've learned that you might not expect. Lessons about leadership on a dark night on an aircraft carrier in the middle of a churning sea. Lessons about the fear you feel with 7 million pounds of thrust exploding underneath you. And most of all, there's an idea out there that astronauts are always perfect. Failure is not an option. Right. That's why I want to take you through some of my life experiences to show you how that's just not true. I believe everyday Regular human failure, if we handle it right, can be one of our greatest opportunities to learn, grow and succeed.

Knowable is accessible on your phone and on the web and each audio course is broken out into individual lessons, usually around 15 minutes long. As a Changelog listener, you can get an annual membership to NOBL for 20% off. Get unlimited access to every noble audio course. Right now, just download the Knowable app or visit knowable FYI and use code changelog. For that 20% discount, we put a link in your show notes for easy click ins, check out Knowable today and start learning from hundreds of top experts from around the world. Once again, that's Knowable FYI Code Changelog.

So this is really interesting to me and I'm as I'm listening to you, I'm trying to think how I'm going to implement. So can you kind of describe some of the good examples of how Green Eye Green AI has been implemented before and any kind of, any kind of guidance? So if I'm, if I'm a practitioner, you've hit on some of the practices, but either go through someone else's example or something that you've described to people because I'm just trying to really make it to where when I walk out of here I want to be able to go ahead and implement that.

Yeah. So I guess I'll talk a little bit about this. So one thing that I mentioned already was performance efficiency trade offs. And I think that the key idea here and one thing that we found when we did this survey that Roy mentioned of papers in our field is that most papers just don't report anything. They don't report any efficiency related metrics at all. Most papers in our field invent some new model or some new loss function, some new training scheme, something like that, and then claim in a table here is our better performance. We beat our baselines. But they don't report for example, training curves or some other measure where you can trade off efficiency and performance. Maybe accuracy could be one measure of performance. An example of this and I guess the first thing that I would say here is what we hope everyone in the research community starts to do and we are seeing this happen now is just report something. Report some measure of how maybe it's going to be the floating point operations to run your model. Maybe it's going to be a training curve, maybe it's going to be the results from your hyperparameter optimization search. Right. So one example of this I can point to is a paper and I think this is, I use this as a positive example of how somebody can report this kind of information. So Roy and I wrote a paper that used early stopping. So we partway processed an example and then potentially had our model stop early. So instead of feeding the example all the way through our model and then coming up with a prediction at the end, we had ways for our model to stop this computation early and make a decision quickly. And this method allowed us to show performance efficiency trade offs, these smooth curves which anyone can then compare against at any point. And what I would hope to see is other work come along and show a better curve rather than just a single point on this performance efficiency trade off. They can report just here's how efficient my model was and here's the performance potentially beating our entire curve or just a single point better along one of those dimensions. In this way, just reporting more information allows others to compete along either of those dimensions or potentially draw a better curve.

So I'm curious, I think a lot of what we've talked about has been focused on like what are ways in which we can still explore this regime of like large models but potentially be responsible about how we're reporting the cost of it and, or how we're allowing others to build on top of what, what we're building. I'm wondering how maybe another side of this fits into this whole discussion which is just playing smaller and more and or more efficient or different models. So I'm thinking of things like recently I was playing around with like Quartz Net, which is this end to end speech recognition model from Nvidia, which is very compact based on these like 1D time separate bulk convolutions. And it's like, like the whole model like on disk is like 90 megabytes or something like that. And like shows like really good performance, almost like comparable or comparable to like these really large speech recognition models. So I'm curious, maybe that also has some advantages in terms of like some of the interpretability things Roy, that you're interested in. Where do you see that this whole regime of new and different more efficient models fitting into this and do you see momentum in that area or good examples in that area as well?

Yeah, I think there's been a lot of, I mean I think the thing that I said a few minutes ago about seeing that we saw very little work that focuses on efficiency. I think in the last couple of years there's been more and more work that focuses on that and we're delighted to see that it probably has nothing.

To do with us.

It's probably Something that would have happened anyway. And I think the main ideas that are being mostly explored are ways to make inference more efficient. And this makes sense, at least in the environmental aspect, but also just in terms of you want to put a speech recognition or an image processing or a text processing machine on your phone and then you need for it to be small in terms of number of parameters or the amount of space it requires or doesn't require much energy, so it doesn't drain your battery and so on. So there has been a lot of effort along these dimensions. And I think the main governing technology there is to train a big model, you know, train it as big as you can and then train another model to imitate this model to some extent or to take them as the large model and get the same performance using fewer resources. There are different techniques of doing that, but that's probably the most common thing that we've seen. What I think is very interesting and people aren't putting that much effort into is to make the other parts of the process more efficient, namely training and what we call model selection, basically hyperparameter tuning or other ways of selecting your best model. And I think this is an exciting direction that relates to the motivation. It's not like there's Jesse think and my thing, we're both excited about both of these motivations and I think this is really one way to improve the ability of the entire community to conduct cutting edge experiments by reducing the cost of these processes.

So in those other parts of our process that you're talking about, I can just imagine, like there have been times, and I will totally confess to this, that like, whether it be hyper parameter tuning or like model selection or something like the logically, the easiest way to go about that sometimes is just to say, oh well, I can have this run for like a week and a half and like go through all these things and you know, there may be a more like a smarter or better efficient way to find the right zone that I should be in, but I can just like get this running and like come back to it in a, in a week and a half or, or whatever. Do you also find that to be sort of like a thing that you're talking to people about and a thing that you're running into is that just sort of like. I don't want to call people lazy.

We're kind of spoiled in that way. That's what I was thinking actually.

Programmers and researchers are often lazy. They have a machine, let's just run.

It for a while.

Yeah, I mean the thing is, this is super common. There absolutely is a trade off between how much time you put in as an engineer or as a researcher, as any kind of practitioner. There's definitely a trade off. You could really carefully narrow down your hyperparameter ranges and then spend less in GPU hours to find some good optimum. Or you could just set it up to be a super broad search, let it run for a week, and it'll take you personally two days, less time to run those experiments of your own hours. Right. The thing is, this happens. Everyone does this. There is some way to often reduce the amount of time that you have to manually engineer something. And another way this can happen is you'll think of some algorithm to say, implement to do inference in your model, and then later you'll be like, oh, you know what, I could make that faster by maybe 5% if I spent a full working day rewriting all of that code. Sometimes it's just not worth it. The key idea, I think, behind our green AI paper is that this happens all the time with people and often we just don't report that. So one analogy that I use is that in our field we don't keep lab notebooks. We just don't record a lot of the experiments that we run and we treat those as negative experiments, experiments that don't show what we're looking for and then we only report the positive experiments at the end. Right? So we just report the single best performance that we found. But with our green AI paper, what we argue is that we should be reporting, even if it's not always the most optimized, the most efficient approach, the best thing that we can do right now is just report something.

It's a really good point there. And I wanted to ask Roy, I wanted to bring you back into it for a moment. One of the things that you say in your paper is you say, finally, we note that the trend of releasing pre trained models publicly is a green success. And we'd like to encourage organizations to continue to release their models in order to save others the cost of retraining them. So, you know, how far can you really get with pre trained models? Do you feel that that will, will do that and is that kind of the way we should get people to start thinking about it? Because it seems like there's certainly a, a training component here in terms of driving people down the right path?

Yeah, that's a great point. I mean, so again, we struggled a lot in the paper when we were writing it, how to not, I mean, what we call red AI kind of, there is a kind of the negative connotation there. But basically I think there's tons of value in these large pre trained models. And definitely once you release them, other people can train models much more efficiently. Because if you build of models like, I don't know if the name is Bert or Bert, I mean a lot to many of the listeners, but I mean these are typical models that are pre trained. I mean some company, in this case Google or Facebook, put a lot of effort into training them and now they release them and other people can take them and use them for their own tasks and the result will be much cheaper than if people train their own model from scratch. So this is definitely something that we encourage companies to do. I say companies because companies are basically the only entities that can afford to do this. And again, our point is that these organizations shouldn't stop training these huge models, but we should be thinking about the negative consequences. And one way to mitigate the negative consequences is to make these models public again, to reduce the overall cost for everyone to run their experiments.

Yeah, so that has a huge benefit for those that are able to use those pre trained models and utilize model hubs and that sort of thing. But of course there's like this element of companies where of course they're driven by money, companies make money and they often want to keep their models proprietary or something like that. But I think also like some of the things you highlighted earlier is that they're in terms of commercial benefit and cost savings. There's also a cost saving element to being able to utilize something that's pre trained and maybe fine tune it and that's a huge saving in labor. Right. But also in utilizing these more efficient or smaller models, like maybe for inferencing, you get less latency, you have less computational costs, all of those things. Do you think there is that sort of commercial or cost based argument to be made to companies?

I think so. There's one thing that we saw recently, there was a citation, I think it was from Nvidia, that claimed about 90% of the cloud cost for machine learning was for inference and only 10 to 20% was for training. So if you can spend a bit extra during the training phase, but end up with a model that's a bit more computationally efficient for inference, then potentially that could lead to savings in terms of the amount of dollars spent renting instances in the cloud or GPU hours for inference, for example. I think that a lot of our focus has been on the research community. So you asked a question about are companies motivated to keep their pre trained models proprietary. While that's true to some extent, my guess is it's hard to know. It's hard for me to know if a company has done that. It's definitely possible. It's almost surely happened that some company has spent a lot of money training a model and then hasn't released it because it's part of their business. At the same time, what we do know about is the research community and this has grown exponentially. Not just the size of our experiments has grown dramatically in recent years, but the number of people in our field and also the number of papers that are written and the size of our conferences. So across we are already seeing such a tremendous growth there. I think it's very worth it to focus on helping save computational costs across inference training, what have you.

Yeah, I guess I think I'm the only person on the call working at a for profit commercial entity and certainly there are times when we aren't releasing that, you know, the way you would in the research community. So maybe I'm kind of curious, you know, would it make sense for us to, you know, yet you still have a group of, of people working in the organization that want to do the right thing always, you know, so they're, you know, they're no different in that way. So maybe still having internal targets for efficiency, kind of like what you talked about earlier and those internal metrics so that even if you aren't publishing them, publishing them for competitive reasons or whatever it may be that you're, you have a set of metrics that you're trying to, to achieve and that might be, that might be something they could spread through the commercial space even when they're not willing to, to do a full release. Is that sound like a reasonable, you know, plan, you know, for those of us who, who do want to strive toward that but maybe don't have the freedom to just release?

Yeah, definitely. I mean people have reached out to us from four crops for profit companies and with similar stories to what you're telling and they want to, you know, they work in a for profit company so they're limited in what they can do, but they want to promote this, they sympathize with the motivation and they want to do the right thing within the scope of their, you know, what they can do inside a company within commercial constraints. Yeah, exactly.

Yeah, I get it.

So, yeah, I mean, you know, as Jesse said, I mean most, we're researchers, we're not part of, I mean Any company is different, I guess, with its own set of norms and rules.

Rules.

And we mostly communicate with the research community. But I mean there's stuff to be done everywhere. Thinking about efficiency, you don't have to persuade anybody that if all other things being equal, if your tool runs twice as fast or takes half the amount of memory, then everybody wins.

Great point.

It's harder when you say, okay, I want to give up a fraction of percent or 1% or 10% and get it to run twice as fast. And there, you know, it's hard to goes into questions of politics and regulations. And then what do these companies, what is the price for them to have expensive models running? Again, more on the environmental side because this doesn't relate to the research community, because it's not open anyway.

Yeah, I think another thing to build on that one thing that we're hoping with our. For example, the track that we have at these upcoming conferences and the conferences that have happened is a place where you can look for research that does directly aim to improve efficiency metrics. So as Roy mentioned earlier, distillation is one approach that's pretty popular about taking a large model and making it smaller and more efficient. There are a ton of ways to do this. So model compression using the lottery ticket hypothesis, or like Roy and I had a model compression paper, there's a lot of ways that people are taking existing work and making it more efficient. And with this track at these conferences or just in general promoting these ideas, hopefully one thing that you can take away from this is a snapshot of ways that you can improve efficiency that have a good track record in the research community.

Awesome. As we kind of close out here, I'm curious since you both have like a very close pulse on the research community in particularly your own areas of research, but also sort of more generally, I'm curious if we were to imagine in the future and there's a world where like green AI is the thing that, that everyone's doing. So some of we've reached some of those goals. What else in the AI research world or maybe like things ways in which people are applying AI. What gets you excited as you look to the future of the industry?

That's a great question. Something that keeps me busy thinking about, thinking about the horizon of where I want to take my work and where would I like to be in 10, 20, 30 years? So I'm excited about a few things. One I think I started with and to taking this amazing technology that does things that are far beyond our reach and we seriously as someone who's been around not a ton of time, but I mean, even five, seven years back, nobody would even imagine that we'd be anywhere close to solving the task that we're currently solving very successfully. And the questions that remain open are, how are we doing this? Are we doing this? Because the models are very good at memorizing and they're just learning everything and are very good at retrieving the information that they've learned. Are they really doing some sort of inference that requires some logic or some, you know, I don't want to use the word thinking, but you know, something that requires some processing, that requires things that we as humans do. And could we generate models that explain why they reached a certain conclusion rather than others? And could we trust? I mean, we obviously can do whatever, we can generate an explanation, but is this explanation faithful? And another thing that gets me excited is to use this technology for all the good things that it can do. And in particular, thinking about doctors nowadays that how can we take things off their plate, allow them to do more of what they're. There's tons of applications of starting from doing better analysis of X rays for radiologists and to transcribe their patient summaries in a more efficient ways and to be able to extract information from that. There are tons of applications here that this technology can be used to make things better for lots of people. So that's things that I'm excited about.

Awesome. Yeah, us too. I know Chris and I both resonate with those points. So what about yourself, Jesse?

There's a lot of things I'm excited about. I think Roy, I mean, things Roy brought up, even just now, I'm like, those are all really cool. I want to work on that stuff too. I think for me, continuing to work in these two pillars of my research so far, which has been reproducibility and efficiency. These are pretty broad categories. So along the efficiency line, one thing that I have been continuing to think about is at least in nlp, what we've seen is larger and larger language models which are pre trained on tremendous amounts of data. And then right now what we've been doing is fine tuning these models, so updating all of the weights in the model so that we can perform well on some downstream task that could be sentiment analysis or some kind of other types of text classification or whatever. My guess is as these models become larger and larger, there's probably going to be some other way that we can apply them to problems that we're interested in. An example of this that has recently been popular is adapters. So that's like adding a small number of parameters to one of these large pre trained models and then only updating that small fraction of the total number of parameters. I think the high level motivation here is that if these models are huge and we want to take one massive pre trained model and adapt it to 100 different tasks, we don't want to have to have 100 different copies of this model, we want to have some smaller fraction. So I think that that is a pretty motivational idea. Exactly what the next big thing in NLP is going to be. The next big idea about how we take our pre trained models and apply them to many different tasks in a relatively efficient way. I'm excited to see what that is. I think one similar idea, one way that we might do that is through probing tasks. So being able to probe our models without updating the weights in them to understand the kinds of inferences that they can make. I think that's a particularly interesting topic. That's very active right now. I've seen too many papers to read just in the last month and a half on trying to probe existing models. And then on the reproducibility side, we've had the reproducibility checklist now used for every submission at I think four conferences. That's a huge success. I'm pretty happy with the way that's worked out. The reproducibility checklist, I guess to give a little more information on that, is a checklist that's designed to remind authors of the kinds of information they should include to make their work reproducible. So it has like, did you include the number of parameters in your model and did you include the size of your data sets, for example? I'm excited and thinking about what we can do next with that information and also with the checklist. So now conferences are adopting it on their own. I've had to advocate in the past reaching out to the conference chairs and saying, hey, I think we should do this. Now conferences have picked it up on their own, which is pretty exciting. So I'm thinking a lot about how we can continue to measure the quality of the research that the community produces at that community wide level and what we can do going forward. What's the next iteration of the checklist going to be, for example? So that's what I'm thinking about.

That's awesome. Yeah, and congrats on the success with that and getting that out there and sort of self propagating at this point. I also agree with you. There's a lot of papers even you've mentioned in this conversation. Too many papers for me to read in a lifetime. There's so much exciting stuff going on. But really appreciate both of you taking time to join us and discuss this really important topic. I hope that people check out your your paper, which will link in our show notes and we'll link a bunch of the other things that Roy and Jesse talked about, so be sure to check those things out and definitely spend some time. Hope our listeners spend some time thinking about this topic and how it influences their workflow and other things. So thank you both and hope to talk to you again soon.

Thank you so much. It was so much fun.

Thanks for having us.

Thanks for coming on the show.

Thank you for listening to Practical AI. If this is your first time, make sure you subscribe so you don't miss a thing. Head to PracticalAI FM to subscribe or find us in Apple Podcasts, Spotify or wherever you listen to podcasts. And if you get value from the show, please do share it with a friend or a colleague. We appreciate you speaking Spreading the Word. Practical AI is hosted by Daniel Whiteneck and Chris Benson, it's produced by Jared Santo and our music is provided by Breakmaster Cylinder. We are brought to you by some awesome sponsors. Shout out to Fastly Linode and launchdarkly. That is our show. We hope you enjoyed it and we'll talk to you again next week.

Sa.