AI in the majority world and model distillation

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by DigitalOcean, the simplest cloud platform out there and we're excited to share. They now offer dedicated virtual droplets and unlike standard droplets which use shared virtual CPU threads, their two performance plans general purp CPU optimized They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, cicd, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers or active front end web servers that need to be full duty CPU all day every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive starting at 40 bucks a month. Learn more and get started for free with a 100 of credit at.co changelog again.co changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're at Practical AIFM and now onto the show.

Welcome to another fully connected episode of Practical AI. In these episodes, Chris and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss some of the latest AI news and trends and we'll dig a little bit into learning resources to help you level up your machine learning game. I'm Daniel Whitenack, I'm a Data scientist with SIL International and my co host is here, Chris Benson, who's a Principal AI strategist at Lockheed Martin. How you doing Chris?

I'm doing great. How's it going today Daniel?

It's going pretty good. Yeah, it's been reasonably normal week in the sense that I've just mostly been at my desk here working on things. But yeah, a lot of exciting NLP and you know, language related stuff coming up this fall and into the spring for me, so I'm excited about that. But your, your week's been a little bit more exciting, I hear.

Well, I'm in London right now as we're recording this and started off doing some stuff with the the Royal Academy of Engineering representing Lockheed Martin.

And yeah, were were you knighted?

I was not knighted and I was so Disappointed that I wasn't. I'm.

Wait, is it. Is it only. I guess it's only possible for English citizens or UK citizens to be knighted. Is that true? I don't know.

I'm not sure. But, you know, I was married to a Brit and my daughter is half Brit, so count. I was hoping that they'd make an exception, but they didn't. I'm going to register a complaint with Buckingham Palace.

But they. They didn't kick you out. It was a good experience.

They didn't kick me out. Did a talk earlier in the week and did a panel and was privileged to meet a lot of really, really, really smart people. And that was really good. I still had to work remotely after that was done because we're about to start a family vacation over here. So I was just kind of waiting for family to arrive and I dashed off to North Wales and climbed Snowdon and then still put in a full work day every day. And so it's been busy. But I'm back in London now and tomorrow is vacation, so all is good, my friend.

Yeah, you're almost there. We gotta. We'll get through this podcast recording and then it's all vacation from there, at least until I talk to you next.

I'm already drinking a Welch beer right now, so I'm getting started.

Okay, Nice. Cool. Well, today on the Fully Connected episode, a few of these episodes in the past, we've kind of dug deep into individual topics. In the last one, we talked about high performance computing, which was great. But today we're just going to take some time and discuss a little bit about some of the trends that we're seeing in the AI community. A few news stories that caught our attention. And we've done this in the past, but every once in a while we like to kind of update things in this way. So to get things started, I just kind of wanted to bring up a trend that I'm definitely seeing, which I'm encouraged by and I think is a really good sign. And that's this, what's seeming to be some good momentum in terms of AI contributions, AI education, AI research activity, all of these things in the majority world. So outside of the US and Europe, in places, countries in Africa or maybe India, Southeast Asia, and where most of the people in the world live. So it's great to see a lot of AI activity starting to happen in these areas. There's definitely still a lot of, you know, room for growth. There's a article that we'll reference in the show, Notes and We'll reference all the articles in the show notes that we talk about, but they kind of look at the publications for the Neurips conference and of course those are still highly dominated by the US mainly, but also European contributors. But you can kind of start to see some other regions of the world contributing. So there's a lot of room for growth. But I've definitely been seeing a lot more attention being placed on education, AI research, AI innovation in other places around the world. I don't know if you've seen this as well, Chris.

I have and it's really good to see. We talk so much about, about the democratization and commoditization of AI in terms of accessibility and so seeing it really blossom all over the world and not just in kind of, you know, US and Europe and kind of key nations around the world, it's really good. I, you know, we talk a lot about, I like seeing it every time, you know, it's in Africa and I know we've talked about in the past and I think we're going to talk about that a bit today. So I'm pretty happy. And I've also noticed it's going into a lot of colleges and universities around the world that are not like top tier strictly. And so instead of everything strictly being, you know, like in the us, you know, Stanford, mit, that kind of thing, there's a lot of second tier universities that are trying to do their thing in it and I'm very encouraged by that. It's really becoming available to everyone.

Yeah, definitely. And I mean there's a lot of problems that are relevant to AI that are probably people from those regions have more domain expertise and more empathy for these sorts of problems around things like translation or things like certain agriculture related applications or other applications that, you know, I'm sure people are working on. But these sorts of problems, the major application of those seems like they would be in places where there is a lot of language diversity or where there is a lot of agriculture. Right. You know, some of the things that I've seen recently are the iclr iclear conference is happening in Ethiopia this year, which it's a big AI research conference, which it's really great to see that happening there. I wish we could be there maybe if there's any organizers listening to this and you want our podcast there, definitely let us know. I'd be up here going to Ethiopia.

Absolutely.

Would you?

Yeah, totally. And I agree and I say this, I'll say it quietly. I'm sitting in London right now and I don't want to see all AI stuff in, you know, New York City and London and, and San Francisco and places like that. You know, it's fantastic to see Ethiopia. I think, I think we should start tracking that a little bit in terms of places you wouldn't expect to see it. That is popping up and give a shout out to people like that.

Yeah, definitely. So I know Google AI, for example, has started opening, I should say opening offices in other places around the world. They just announced, I forget when this announcement was. It was just in the last week that I saw it at least that they're opening up an office in Bangalore. In Bangalore, India. But also they have offices in Ghana and Beijing. And then also we're mentioning Africa. I know there's a series of conferences and events and workshops, you know, called the Deep Learning in Daba that's happening. There's. I've seen other things that are being sponsored by companies like Facebook, Google, Microsoft and other sort of large tech entities that are happening there. There was a Southeast Asia machine Learning school. There was an AI for India summit that Facebook put on in India. I think that was actually also in Bangalore. It seems like a lot of these tech companies are really interested in developing AI talent and AI expertise in these majority world countries. And I have my own thoughts about why that might be taking place. But do you have any thoughts on sort of why tech company like Google or Microsoft would be interested in that?

Well, I mean, I think part of it is it's really hard to find great talent. And so I think having a diversity of your, in terms of your search for the people that are interested and capable of doing this, it makes sense to go to third world countries and take a bunch of interested people and bring them into the fold. I love that. I'm looking at the Indabo website from a few weeks ago when they had the conference. And to your point, the sponsor list is a who's who. It's DeepMind, Microsoft, Google, Facebook, IBM, Apple. I mean it's the same tech companies that you're going to see sponsoring stuff in San Francisco and New York and London and such. And so I love seeing that attention and I would love to see in this case, since we're talking about Africa, Africa being able to really get a great AI community behind it. And while they're at it, they should listen to our podcast. Yeah.

And we should get involved as we can for sure. I mean, I definitely agree with you. I think that the talent is one side of things. I also think that hopefully These companies are starting to realize, like I mentioned, that some of the problems that they're really putting a lot of focus into, like Facebook is putting a ton of focus into machine translation and language tech and really the, you know, the sort of real world experience and domain knowledge that people from Africa or Southeast Asia could bring to those sorts of efforts, I think you're just going to end up with better results by using expertise that's, that's rooted and has experience in those areas. And of course, you know, AI I think is going to be kind of ubiquitous in the software stack as we move forward. And so as we try to build up, you know, software engineering in general in these, in these different areas and education in general, I think it definitely needs to be a part of it. I wanted to share too. Recently in July, I was in Singapore and had the chance to stop by the AI Singapore offices, which you might have heard of programs like this. So it's kind of like the Prime Minister's office in Singapore says Singapore needs to be a leader in AI. How are we going to do that, how are we going to develop local talent in AI? How are we going to contribute to research, etc. Etc. And they established this AI Singapore organization which is associated with the Prime Minister's office and their job is to basically figure that out. I have to say I was, I was super impressed with the program that they've put together there. It seems to be run like a well oiled machine. So companies can come to AI Singapore and they basically say, okay, if you have a problem that's related to AI, let's partner together to solve that. What we're going to do is we're going to form a team in AI Singapore which includes, you know, really top notch mentors in AI and researchers in AI, paired with AI apprentices that they're training up and your engineers from your company. So basically everybody's leveling up in AI at the same time and their focus is really not just to kind of learn interesting things, but they're working on real world problems that can be solved and they really want to take on problems that will be pushed into production in commercial entities. So they have this whole program around it. And I have to say I was super impressed with the talent that was there and the program that they have going on. So it's just cool to see that sort of thing happen. And I shouldn't have been shocked, but I was kind of a little bit shocked that this was going on in Singapore and at such a great level. And I had no idea about it.

So is it accurate to say that it's sort of running like an AI incubator the way you see incubators in the US where they're nonprofits and university affiliated organizations that are taking the talent in, they're just doing it instead of being around a university setting specifically, they're doing it out of the Prime Minister's office. Is that a fair way of assessing?

Yeah, it's not unrelated to that sort of incubator idea, but it's really more maybe an accelerator or something like that paired with the educational piece is there? Because what happens is every year they have, you know, engineers and whoever from Singapore apply to become AI apprentices in the program. And I think they said they were like, you know, it's really competitive, I'm probably going to get the numbers wrong. But there were like 800 people applied and 20 get in. Right. So it's really competitive. And so you got these top notch applicants and they're trading them up over this sort of nine month period to be AI engineers. And so they go through a little bit of training, but then they're also kind of their capstone or the project that they work on is an actual problem within a company, tech company that they're working with. And so they sort of form this collaborative team with mentors and the apprentices and the industry company to actually solve a problem that will go into production. So it's kind of existing companies leveling up their AI expertise while at the same time, you know, developing AI talent within Singapore. It's pretty cool.

Do you think you're going to see more of these popping up associated with various nation states around the world? Do you think, do you think this is going to be a kind of a common blueprint that we'll see?

Yeah, I certainly hope so. They kind of seem to have this down and I kind of actually shocked because most of the times when I think about like internship or sort of accelerator programs or project focused program partnerships between academia and industry, a lot of times those seem like they're run really poorly. And I think what I was shocked by was this was run really well. So I think if other countries are really serious about this, I think it is a model that we might see more of. I think it's hard to get right for sure because you know, how many times have we seen, oh, you know, this company is going to work with this university or there's going to be like this center of excellence at a university that we're going to work with certain companies from industry, and then, you know, some stuff happens, but it's not really that impactful. So I hope that these sorts of things are a little bit more impactful.

Foreign.

Practitioners, Jared here, wanting to let you know that Changelog will be at all things open on October 14th and 15th. We're hosting a live JS party on stage, and as a special thanks from the organizers, we're giving away five free passes to the conference. All you have to do is tweet. I want a free pass to All Things Open because state your reason and mention changelog or practical aifm. So we see it and we will DM you if you win. Okay, that's all for me. Let's get back into.

Was definitely great to talk about some of the things that have been going on in parts of the world that we're not currently in, but there's certainly a lot of AI news coming from the rest of the world as well. And I think that you had something you wanted to highlight that you saw in. Was it from MIT or where was it from?

Yeah, I saw it on phys.org and so it's an article that I ran across and called Artificial Intelligence Probes. Dark Matter in the Universe. And so.

Yeah, from ETH Zurich.

Yep.

Sorry, I got that wrong.

Correct. And so I know that you're a physicist by background, and just as an amateur, I love physics. And so it certainly caught my attention. And it was interesting that they were trying to explain it by drawing an analogy with facial recognition in terms of how they're using models to scan, you know, scan the universe and try to understand what both dark matter and dark energy are. And for listeners who may not be familiar, you know, the dark matter exerts gravity on the universe, and you can measure that, but we can't see it. It's kind of like gravity coming from an invisible source. And, Daniel, if I. If I mess any of this up, you should correct me.

No, you're. You're doing great. Yeah.

And also the universe is expanding, constantly accelerating in this expansion. And so that's dark energy. And once again, we cannot see that source. And so those are two of the great mysteries in physics at this point, is trying to identify and, and understand what we're observing. And so seeing that, that they're using models to try to. To recognize those features and find the patterns that maybe otherwise we're not seeing. That was pretty cool.

Yeah, definitely. I think that there's a sort of general trend in science where these sorts of AI techniques are being Applied that are. That are making an impact. And I've mentioned a couple times on the podcast where machine learning started to make an impact on the field that I was studying when I was in grad school. But I think the pattern was similar there. And that's the sense that, you know, we have these experimental observations which are kind of, you know, they're rooted in reality, and we have a little bit of knowledge about how things work and, you know, constants and other things and certain laws that shouldn't be violated. So constraints, essentially. But the sort of relationship between input and output could be incredibly complicated to write down in terms of equations or maybe computationally too expensive. So the problems that we were working on were, you know, we know that this atom or molecule has this many particles in it, right? Electrons and neutrons. But to kind of write down the equations and actually make the computations about how all these things work together, it's actually computationally infeasible to do that, you know, and just by the equations that we write down, and in this case with the dark matter, you know, we know some of these constraints, we know the experimental observations, but we're not able to sort of maybe write down, well, the statistics like that they're talking about that govern these things. And so kind of plugging in a neural network into that gap and having it try to learn some of the features that are important to some input. Output. Whether that's input, in this case input, it sounds like these experimental observations and output cosmological constants and other things, or in the case of atoms and molecules inputting, like, geometries or numbers of particles and outputting energies. Putting a neural network into that. That gap where things are really hard to model can make a lot of sense, I think.

Yeah, I mentioned that they draw that analogy, which they talk about how Facebook uses its algorithms to find eyes, mouth, and ears and images, and that they're looking for these telltale signs of dark matter and dark energy that they're basically looking for light bending, you know, and so as light is bent by the gravitational influence, I'm assuming that they're using convolutional neural networks, although they don't explicitly say that to try to notice the subtleties in terms of identifying the relationships. But it's just really interesting to see it in being used in this way. I'm always fascinated to see all the different use cases across industry as we see this becoming more and more pervasive over time.

Yeah, data science for science. I think that's kind of in Vogue right now.

I think it's a delightful redundancy of the word science, isn't it?

Yeah, very, very meta. Yeah. So speaking of language, you know, listeners always know I'm keeping up with language related things. So maybe you're out there, you're listening, you're like, oh, Daniel's going to share another language related thing. I'm tired of that.

Don't worry, we're going to do an episode where you're going to share all that with us soon. So you're not getting out of it that easy.

Cool. Well, I think that this kind of has a more general angle on it, so not just language. So hopefully our listeners are okay with that. But I was really intrigued by this recent thing that Hugging Face released. So if you don't remember, the CEO of Hugging Face was on the podcast here talking about social AI and conversational AI, and that was a great episode. We'll link it in our show notes. But they've kind of really been working on these large scale language models. You might have heard of Bert or Elmo or GPT2 from OpenAI. And so with this most recent release or one of their recent releases and publications, they kind of took a different angle on it and it's called Distilbert. And so what they said was, okay, the trend in these language models is that they keep getting larger and larger and they're trained on more and more data. And in fact, one of the references they give is a latest model from Facebook AI that was trained on 160 gigabytes of text. Which that might not seem like a lot for those that do images or videos, but if you think about 160 gigabytes of text, that's a lot of text, raw text data, that's an enormous amount of text data. So they were motivated by the fact of looking at those things and then saying, okay, well, if we actually want to use those types of models in production, how do we do that under the sort of low latency constraints where we might want to make a lot of predictions in a very short period of time, like in a microservice or something like that? Also, how would we run those types of models on a smartphone where there's obviously constraints around energy efficiency, you know, maybe memory constraints, and also just generally the environmental cost of running large models like this is huge in terms of the computing requirements for them. And so they took this and said, what would it take to create a smaller, faster, cheaper, lighter version of Bert, which is one of these large scale language models? And they ended up doing that and this is what they're calling Distill bert. And it actually has very small penalties in terms of evaluation, but it's, it's smaller, faster, cheaper, lighter in comparison. They have a little graph of how many millions of parameters various models have and the latest, so there's an Nvidia model language model that has like 8300 million parameters and distilbert has 66 million. So significant reduction in size. But only distilbert still maintains 95% of Bert's performance on language understanding benchmarks like glue, which I think for such a reduction in size with only a very small, in some cases negligible penalty in performance, it's just really encouraging to see.

So since we're fortunate that you're kind of an expert in this topic to some degree, I've seen lately a lot of different articles about different types of compression and different types of architecture construction in an effort to make it more performant. Do you have any insight into, into how they might be approaching that or even if not, how would you tend to, to do that and what kind of benefits can you get out and what kind of applications might it enable?

Yeah, definitely. I not an expert in all areas of this field. There's a lot of different techniques used here. Of course, there's a lot of, a lot of people working in this area. You know, intel and Google and others are working to kind of shrink down models, to put them on smartphones or even microcontrollers and other things. And so there's a lot of work going on there. Some of the methods kind of fall under, I think, what's called pruning, which is kind of like cutting out parts of your network that might not be having an impact. Hopefully. I'm saying that somewhat right as I'm not an expert on that.

I had a conversation with a professor at Georgia State who was, who was doing a version of that for compression and, and so yeah, keep going. I wasn't trying to cut you off.

Yeah, no, so that's like you could kind of think about that technique as a, like you train model and then afterwards you go through this sort of optimization or compilation which kind of prunes things out or makes the model smaller. So it's kind of like a post processing thing, if that makes sense.

Do you think that'll be common in terms of, as a technique in this area? You know, with you doing NLP all the time and you know, is this going to be a standard part of NLP deployment going forward?

Yeah, I think it'll probably, at least based on my understanding, it'll depend on the type of model, the type of task, and also the type of target architecture. In terms of like actually how small do you need to get? I think that some of the goals of the Hugging Face team were to get the model small enough to where they could run it efficiently in production and maybe on smartphones. Right. Which still are actually pretty computationally powerful if we at least compare them to like microcontrollers. Sure, but so they use this technique called knowledge distillation and that's why the model is called Distilbert or Distill bert.

Which has to be a knockoff Dilbert.

Right. They should have come up with a logo as such. Yeah. So this model you might have heard of sort of teacher student training models. And the idea is, I think again, you know, please our listeners correct me if I'm wrong, but I think the basic idea is that you have a sort of teacher model is a larger scale model, maybe like full bert, let's say. And then you have a smaller model that is supervised during training by the larger model. So you try to get as close as you can to the larger model's performance and output distribution using this kind of teacher student supervision. And that's kind of interesting because you kind of do a full scale training on a larger model so that that still happens, but maybe it doesn't happen over and over and over. You kind of are able to train these, these smaller models to still get most of the performance out of the tasks that you're concerned with. And so I think this was the type of methodology that the Hugging Face employed. Hugging Faces is a very open source focused company. And so in their blog post even they show some of the Pytorch code and illustrate how to do this in Pytorch. So if you're interested in knowledge distillation and teacher student models and want to actually get your hands dirty trying out some of these things, maybe trying your own distillation, then that would be a good place to start, I think, because you could look at some hands on examples.

This episode is brought to you by Brave. The Brave team is on a mission to fix the web by building an open source, privacy focused and performance oriented browser. Browse the web up to eight times faster than Chrome and Safari, block ads and trackers by default and reward your favorite creators with the built in basic attention token. Yes, you heard that right. A real world use case for blockchain. Download Brave for free using the link in the show notes and give Tipping a try on changelog.com.

Okay, so I found an article that was interesting to me from the title, but as I started reading through it, I realized that I actually know one of the authors. It's an article called Three people Centered Design Principles for Deep Learning. And it's by Dr. David Bray and R. Ray Wang, and Dr. David Bray and I both used to work at the same company together called intelonet many years ago and we've kind of loosely kept in touch.

Yeah, so that, that's, that's super cool and I'm glad that you like, when you said it out loud, it made so much more sense to me because when I first read the title, I was thinking like three people centered design. I think how do you center design around three people? I guess sometimes you're designing things for three people, you know, interacting kind of thing. So my mind was totally in the, in the wrong direction on that. But thanks for clearing that up.

No, no problem. Actually, when I, just a minute ago when started saying it, I started saying it that way in error as well and corrected myself, if you notice. So, yeah, a little bit of an awkward title there, but very interesting article. And a lot of that from my standpoint has to do with the fact that I'm a big advocate of keeping people and users at the center of technology. And that's not an AI specific principle. You know, it's used across many different types of development processes and technologies. And I liked seeing this AI centered piece. And I. And when I go out and do keynotes and things like that around AI, it's one of the points I'm often making. So I was, I was kind of delighted to see it. And he kind of starts off talking about in deep learning and how you think about the outcome first with the intention of kind of avoiding bias in your process, which is very easy to allow to happen. As we all know, it's probably the most common problem in deep learning that we all face with our data, but how easy it is to get to potentially a bad outcome. And so by kind of keeping your people centered outcome at the center of your process, he kind of says you're more likely to get an outcome and better training than you would otherwise get. And so the three kind of keys that he's talking about here is transparency. And he's talking about the fact that you really need to understand what you're trying to get to and be very clear with what your intentions are in the training and make sure that your data is oriented on the outcome that you want, and making sure that the process of doing the model training is obvious in terms of what your inputs are to get your output as possible so that no mistakes are made. And the second point is explainability. And this is obviously a huge area inside AI research in terms of being able to understand how the inferences that a model is making, how those inferences are being reached. And so that's certainly in the industry I'm in, where we have a lot of autonomy in terms of autonomous vehicles. You know, being able to put people's lives in that and being able to explain how your model is getting that is kind of a key to people having confidence in that. And then the last thing is reversibility and understanding how you reverse out of a model about what it knows. It's kind of tied to explainability. So you're really looking between transparency, explainability, and reversibility. You're really looking at something where your outcome is a little bit less black box, it's a little bit less mysterious, and you have kind of a map on how you're working your way through the network. Obviously, there are limitations on what we can do today in each of those areas because obviously there's some fairly significant research going on. But it's a good focus to try to kind of know that that's the direction you're going. And so I'm looking forward to seeing where we go. Have you come across people centered design, design principles in the past? Daniel?

Yeah, I definitely have. I mean, I wouldn't say I haven't had any formal introduction to the topic, but it's definitely come up in a lot of teams that I've worked on and different organizations. And I think that it's, you know, it's a, it's an important piece. I just remember being in a meeting just the other day when one of my supervisors, we were talking about, you know, the topic just came about, you know, in general, what is the purpose of a commercial entity? And, you know, the idea that, oh, it's, it's to make profit came up, at least that's part of it. And, you know, he was saying, well, some people might think about it that way, right? But I think that whether you're a kind of social good company or organization, nonprofit or a commercial entity, ultimately you're wanting to satisfy customers, right? And that should be at the very forefront of your design, of your product, of, of how you go about your business, that you're wanting to satisfy your customer or your user. In this article, I was kind of wrestling and trying to figure out this idea of reversibility because initially when I read that, I immediately thought of reproducibility, which is something we've talked a lot about on the podcast and had an episode where we talked to one of the founders of Pachyderm about data versioning and such. But reversibility, it seems. It seems very different. So I was kind of interested to read a little bit more about that.

Yeah, that caught my attention as well. And I also went to reproducibility initially in my head and went, nope, that's not what he's saying. So it was interesting to see. I'm kind of curious to see. I think of the three concepts that he talks about, that's the one that I probably need to ramp up on a little bit more and understand how it would be utilized here. But at the end of the day, I think the intention is good. It's funny, I have a personal tie a little bit to this interest, and that is you recently interviewed me on one of our episodes about high performance computing as it relates to AI. And I can say that in the effort that we did at Lockheed Martin and are still doing, that aspect of really centering on the person that is building the model and on those outcomes that you're trying to achieve was really one of the core design principles that we built into our effort to the point of it was every bit as important to do that for the ability of getting to the outcomes that we needed and need as quickly as possible with very high fidelity. And so anyway, when I saw this article, I was pretty interested and maybe at some point, maybe Dr. David Bray might come onto the podcast and tell us a little bit more about it. And I think that would be an interesting episode.

Yeah, I definitely would be interested in hearing more about some of the details that he goes into. So he also talks about creating data advocates, which I thought was an interesting idea. And also a huge piece of this, which I think is important and was mentioned by one of our guests. Lindsey Zoolaga from HireVue is really putting a lot of effort into mindful monitoring systems to test data sets for biases. So if I remember right, I think Lindsay was talking, and this probably intersects there where she was talking about, well, it's good if you put in some effort into thinking about your users and thinking about bias when you're training AI models, but a lot of things drift over time, and the performance of things drift over time. Your user base might change in different ways. And so really putting in a monitoring system that is actually monitoring your online models to really judge whether you're actually dealing with any sort of bias in the input, or if your model is, is all of a sudden generating predictions that are biased between two different groups or, or something like that. So I think that's a really important piece of the puzzle. And I'd love to hear more from the author here about the, the types of monitoring systems that he has in mind. He does even have like a, a chart for the mindful monitoring system for AI, which I think is kind of interesting. I'd like to hear more about that.

Yeah, I would too. Do you. I'm just curious, just from your own personal experience, how do you, when you're thinking about bias in the data that you're working with, and however you may choose to monitor, to try to address it ahead of time, how do you approach that? It's a little bit of a tangent, but if there is one thing that every data scientist that works in AI is going to contend with, it's that I'd love to know how you approach bias and what, what are some of the processes or, you know, exercises you do to try to eliminate bias that might produce a bad outcome for you.

Yeah, I think, I mean, right now I'm working on a lot of language and voice tech, and I think people, when they think of bias, they always think of some, some sort of like, discrimination against certain groups, maybe like based on race or gender or whatever it is. So I don't think it always has to be that side of things. But of course we should be aware of that side of things. I mean, it could just be as simple as, like, I'm trying to translate this piece of text into Hindi. But in my training data, all I had was news data. I only had politics data. I didn't have any sports data. Right. And so it could just be a simple bias in that data set in terms of what it's been exposed to. But also, you know, there's certainly cases where I think marrying that sort of mindset with a mindset geared towards your users can be really powerful. So especially with voice, you want your voice system to work equally well for men and women, right? You want it to work equally well for those that are highly educated and not highly educated, maybe, or those with certain accents and not certain accents or certain regions, certain dialects or not dialect. And so the only way that you're going to be able to be able to do well in that scenario is if you, for one, try to have diversity in your training data when you initially put that system together. But also in terms of monitoring, I mean, one thing is you may never be able to anticipate all the types of people that are going to interact with your system. Right. So anticipating that in advance and putting a monitoring system in place where you could actually tell, oh, in these scenarios, our system isn't doing well based on our feedback we're getting or based on some metric that we're measuring. And if you look further into that, you could identify certain groups that are using your system that you just didn't expect before. And so now you should circle back and integrate that into your training data. So I think it's a cycle and you have to think about it both in production, in training, and kind of this feedback in between the two.

Gotcha. Okay, thanks for going there.

Yeah, definitely. Well, I appreciate you coming up with some, some good articles this week, Chris. Before we head out, like in all of our fully connected episodes, we like to end with just a couple, at least one learning resource that you can use as you're trying to learn more about the most recent trends in AI and kind of level up your skills. So the one that we wanted to highlight this week, which I've just really enjoyed over the past year, is called, and we may have even highlighted this before, I'm not sure, but it's worth highlighting again. It's called Papers with Code. So if you just go to paperswithcode.com it's a site that is essentially what it is named. It's Papers with Code. So it's AI research papers with the links to the code of the implementation on GitHub and in some cases a sort of ranking on various tasks. So on the, on the front page if you go to this site, it'll show trending research. At the very top of the trending research right now is Pytorch Transformers, which is not a, not a huge surprise there since hugging face is killing it. But you can click on Pytorch Transformers on that paper. It's going to take you right to the GitHub repo of the implementation. But you could also click on the paper, right? And it's going to take you to the paper for that. And you can click on some of the papers are tagged with little things like oh, this model or this paper is state of the art in common sense reasoning or something with a certain data set. And so they've also got these pages of what are the most state of the art Papers with code for X task. Like you know, reading comprehension or question answering or, you know, sentiment analysis or these different things. So just an overall really well put together site and something where for example, the other day I wanted to know what are people doing in terms of sign language recognition and who's doing the best stuff. And I was able to get just a few leads on this site that led me in the right direction. So I find it really useful.

Yeah, we have. I do remember that we have highlighted it before, but I also love it just like you do. It is definitely worth highlighting again. And actually as I'm looking at this right now, the third one is one that I'm going to dive into as soon as we're done recording this podcast, which is Deep, a Generative Adversarial Network for Face Anonymization because we had an episode recently on Deep Fakes and so I'm looking forward to diving into that one and see what they have. But thank you for highlighting this site again. It's a fantastic one and after a little while we might even need to do it one more time.

Yeah, for sure. I think it's a good reminder and a lot of things have been added over time on the site that have made it really useful. Well, I appreciate you taking time before your vacation to talk through a little bit of AI stuff, Chris.

I wouldn't miss it. This is fun stuff to me. As far as I'm concerned, vacation's already started. This is part of it. So yeah, I mean this is great. I always love our conversations. We always love to get the feedback that our listeners give us. We talk to people on Slack because we have our Slack community. We talk to people on LinkedIn, we talk on Twitter. And a lot of what this show is about is coming from engaged listeners who ask a question or say, hey, I would really love to hear more about this. And so I really hope that everyone keeps engaging us on this and let us know what you're interested in so that it's part of what makes this fun.

Yep, definitely do. And have a. Have a great vacation, Chris. We'll talk to you soon.

Thanks a lot. Take care, Daniel.

All right, thank you for tuning into this episode of Practically AI. If you enjoyed this show, do us a favor, go on itunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do, share this show with a friend. If you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more@fastly.com and we catch our errors before our users do here at changelog because of rollbar. Check them out at robot.com changelog and we're hosted on Linode Cloud Servers. Head to linode.com changelog check them out. Support this show this episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder and you can find more shows just like this at changelaw. Com. When you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.