BERT: one NLP model to rule them all

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@robar.com and we're hosted on Linode servers. Head to linode.com changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning, data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're @practical aifm. And now onto the show.

Well, welcome to a fully connected episode of Practical AI where Chris, my co host and I will keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss some of the latest AI news and then we'll dig into some learning resources to, to help you and us level up our machine learning game. So Chris, how's it going?

Going great. How are you doing, Daniel?

Doing well. Getting ready for American Thanksgiving.

Yeah, it's as we're recording this, it is the day before Thanksgiving for us and so looking forward to, to overstuffing myself tomorrow and then. And then worrying about how I'm going to lose the weight thereafter.

Yeah, looking forward to some tofurky for sure.

Exactly.

What have you been seeing in terms of AI news recently, Chris?

Well, actually, the first thing that caught my eye was something that you had, had shared with us the last time we were talking about this stuff. And that is we were, we were talking about artwork in the world of AI and there was a, a particular piece of art that you discussed, and that was where a generative adversarial network had been used to generate this piece of kind of pe. Period art. I'm not an art person, so I'm not gonna do that.

Looked like a portrait of a guy.

Yeah, it was called Portrait of Edmund Bellamy, if I'm saying that correctly. And it was a, it's a fictional person from a fictional family. When you were telling us about it last, it hadn't gone to auction yet, but Christie's was going to, and they were expecting it to raise somewhere between 7 and $10,000 US. A few days after our episode went live, it actually went to auction and it ended up selling for 4, $432,500 U.S. boom. Boom.

That's crazy.

Crazy. So I mean. And it really.

Wait, wasn't it. Yeah. So they were saying that it was going to sell for like seven to ten grand, right? Yeah, Something like that.

Yeah, it was like it was 45 time multiple on what they thought. And you know, and so we, we had been looking at it with interest, you know, in the AI community, but I think the entire art world got rocked by that one because suddenly, suddenly. Yeah, you're major contender from, from the, of, you know, high value art going. So I think it's something that a lot of people that are not traditionally thinking about AI are having to digest and realize that the world is changing.

Yeah, I think people are going to have to start or are going to stop going to Chicago Institute of Art and start going to MIT or something to go into art.

Yeah, it was kind of funny. A few days after that I host a meetup in Atlanta called the Atlanta Deep Learning Meetup. And I know I've mentioned it before, but we actually had a Generative Adversarial Network tutorial last month where we had an expert named Reza Katabi come and show us. And it was funny. He came in, he goes, you know, I had this little project, we got all coded as we go along the way, but I've changed my mind. We're going to try to build some artwork in this session and we'll split the proceeds if it makes enough money. And so it was just funny because, you know.

Sounds good. Yeah, yeah, that's great.

And there's been some other, some other big news, not necessarily art related. What have you seen lately?

Yeah, so I don't know about you. I'm on Twitter. That's where I hear about a lot of things. And it seems like to me, and I don't know if you've seen this same thing. Let me know if you have. But it seems like every other AI related tweet that I'm seeing, at least in the people that I follow, is about natural language processing.

Oh yeah.

So like over the past, I would say like I don't know, like three weeks to a month. It seems like there's just been a steady rise in all things like neural nets and natural language. Have you been seeing the same thing?

I sure have. And I think I know where you're going with that because there was a particular thing announced that I'll let you lead into that has really caused a lot of interest in the last few weeks.

Yeah, you guess where I'm going. So there's this new model out, so a pre trained model called Bert from Google. So that's a new approach to pre trained natural language processing which we can talk about here in a second. But there's actually been like I've seen even yesterday, I think it was yesterday, this HTML model from Hugging Face, which is pretty incredible. Take a look at that. If you haven't seen it by HTML, it's not meaning the HTML of the web, but like a multitask learning model. And I'm sorry, I now I'm saying I'm getting confused even with the acronym. So it's hmtl. Right. So hierarchical multitask learning, not HTML. So there's a confusion there.

You know, you just, you just disappointed an entire world of front end developers who thought they just now had their way into machine learning.

Their way into. Yeah, no, so hmtl. So I saw that yesterday. There's also like I've seen Elmo, which I think came from the Allen Institute. And also there was this challenge. So one of the challenges at the now rebranded Neurips conference, which was a much needed rebranding but now they had a competition that's actually, it's kind of in the schedule for the presentations phase I think now around chatbots and dialogue systems. And so it seems like, at least from my perspective, all things with neural nets these days are like with natural language.

Yeah, it's kind of funny. We go through these waves, you know, for a long time everything seemed to be about computer vision and all the different convolutional variants that came out and capsule nets. And you're right, there hasn't been as much in the news lately, but with BERT being released, the NLP world is just on fire right now.

Yeah, for sure. So let's. Actually I'd love to, if you're willing to kind of dig in a little bit. I'd love to dig into exactly, you know what BERT is. I'm still learning about it. So I'll confess, you know, as we get into this conversation, please connect with us on our Slack team and our LinkedIn page of Practical AI. You can go to changelog.com PracticalAI and join our Slack team. But I would love to hear if I say anything that's not right, I'm kind of learning about these things as I go. So we'd love to hear your perspective on these things as well. So keep us informed in that way. But my understanding of what BERT is is the goal is to create this kind of pre trained NLP model or pre trained language model. So some of this terminology is new to me, like I mentioned, but in my understanding what they're trying to do here is create an encoder that will be pre trained that you could utilize for various natural language tasks. So for example, like sentiment analysis or question answering or named entity recognition. These are all kind of natural language processing tasks. And so their goal is to create this pre trained encoder that will essentially kind of be a act as a language model or a model that understands the kind of structuring of words and the context of those that can be utilized as the first bit of other models for these other sorts of tasks.

Yeah.

Is that kind of. Chris, correct me if I'm wrong, am I on the complete wrong track here?

No, no, I think you're right. I think they have some specific terminology they use. I think they call that a transformer. And the transformer is learning the contextual relation between words and text and then it has two separate pieces to it. One is an encoder that's reading the text input and then one's a decoder that is producing the prediction, you know, for whatever task you're applying it to. So I think the. When you combine the encoder and decoder, they're calling that a transformer. Yeah, but you. I think everything you said was accurate. Yeah.

And I think what they're saying, because this isn't the transformer model, I guess has been around for a while and we'll link to the info about that in our notes. But this. So the BERT model, which again has come out of Google, so it stands for bidirectional encoder representations from Transformers. So essentially this is like you just mentioned, Chris, this is based on the transformer model. And it's kind of like you mentioned, in the transformer model there's an encoder and a decoder level because they're trying to do a specific one or more specific tasks. In this case, they're kind of basing this BERT model on the encoder piece of that I say transformer.

Thanks for the clarification there.

Yeah, so it took me a second to get there because it is confusing with the.

It's a lot to digest. There's a lot of information that's been pouring out and I know both of us have been going through some of the different articles and stuff that kind of break it down. So definitely a learning task and process for us.

Yeah, yeah, for sure. And my understanding is that it's based on this transformer encoder, which is kind of unique amongst encoders, because when you think about trying to understand language like the context of a word in a sentence, you can think about it directionally, like moving forward in the sentence and Based on the kind of forward direction getting the context of a word. But actually the transformer in this case is. Well, they call it bi directional, but in my understanding it's really like non directional because it considers all of the text surrounding a word at this, you know, as it's determining the context of a word. So it's not directional in the sense of like going forward through a sentence.

Yeah, that's my understanding. I've seen it described in different people blogging about it as either non directional, as you put it. I've seen it as directional in either way, or bidirectional. I think you have a choice in how you're doing it. And the masking of the word that you're building the context around is pretty key.

Yeah, yeah, definitely. You mentioned the tasks. I think that one of the. Also the key features here is that it's the bidirectional encoder representation. So they're creating this kind of context for the. For language. But in order to do that, they have to kind of decide about what tasks will help them or what tasks or predictions will help them get the best understanding of language or create the best kind of encoder or language model, like they're calling it. And in this case, they're actually using multiple tasks to do that. They're using one task, like you mentioned, which is kind of like a masking of words. Like in a sentence, they'll kind of remove certain words and have them train the model with the encoder to actually kind of fill in those words. The other task that they're doing is next sentence prediction, which is like, given two sentences, can you tell if one of the sentences actually the next sentence that comes after the other sentence?

Yep, I agree. One of the things that it occurred to me we probably should do is kind of talk about, you know, what encoding and decoding is. Encoding is where you're actually taking your input and putting it into the sequencing. And by the way, I found this on Quora for the audience. Daniel and I are googling this stuff just like you are. You know, we're all learning as we go. And encoding and decoding is obviously a common task in a lot of neural network architectures, but putting it into sequence and then decoding is where you're actually getting the output that you're going to use on that. Also wanted to note we've kind of not mentioned that there's really two stages to Bert, and that's important because they're for different purposes. There's a pre training stage and then there's a fine tuning stage and the pre training stage is very expensive. It takes a lot of resource. I think in what I'm looking at here they talk about it takes four days on a 4 to 16 cloud TPU system just to get through.

Yeah, a lot of that's some crazy stuff.

Lot of processing.

I have the thing pulled up right now with GCP's cost and that turns out to be around like well at least with the number I'm seeing around 7K US dollars, so 7,000 US dollars in TPU costs which of course they're Google and I guess they don't spend that because it's their own cloud but significant effort.

My wife would not like it if I did that for a weekend project. I would get in trouble for that, spending that kind of money. But fortunately Google has put out a whole bunch of pre trained models. You know, recognizing the expense of that. They have, they have helped us, all those of us who are going to be trying to apply this technology. They have a good starting point and really when you're deploying it into your own application, the fine tuning, which is an inexpensive thing, doesn't require nearly as much processing is really where you're going to be focusing. So you'll be able to go and find a pre trained model hopefully maybe or maybe not even need to make tweaks to it. They mentioned that there was very little adjustment needing to be made for different use cases and then do the fine tuning for your own specific which is inexpensive and something that I probably could do on the weekend without getting in trouble.

Nice. Yeah. So correct me if I'm wrong Chris. So I'm trying to think about like because I actually have potentially a couple use cases that I have in the back of my mind for this and correct me if I'm wrong when I'm thinking about like how one would go about this. But in my understanding a best use case for me to use BERT is if I have some natural language processing task. Let's say I'm trying to identify certain entities in text like named entity recognition. What I could do is, is take a pre trained bert. I don't know if that's the proper way to say that, but that works for us. Pre trained bert. Sorry to any of you out there that are named BERT and this is confusing for but I would take a pre trained BERT which Google has spent much, much time training and many update steps and lots and lots of data and so they've developed this BERT and what bert's going to do is allow me to put in sequences of words and BERT will then output sequences of vector representations of those words. Also give kind of a context within the language model of bert and then I could kind of bolt onto that encoder layer some classification task or some other sort of task. In my case, maybe it would be named entity recognition. And because BERT is so good at understanding the context of language, actually the update for me to actually do one of these tasks like named entity recognition or, or question answering or something is like you said, fairly inexpensive. So I'm utilizing all of the expertise that has been built into Google's model and just adding on the little piece that makes it particular to my use case. And so the first thing that is pre training. The second thing is fine tuning. Is that right?

I think that was a great explanation and that is consistent with my understand. It's. You know, BERT is really. The way I'm reading it is BERT is really to be embedded into a larger architecture to where you get this incredible capability for maybe not for free, but at low cost relative to having to figure out how to do it yourself or use a lesser technology. So from my standpoint, I think this is another great step where Google in this case is providing what would otherwise be a very challenging specific task in a larger architecture, and they're helping us do that almost like a software component in a larger software system.

I think that there's kind of two threads that I see running through this that are also kind of, you know, hugely impactful, I think in the industry in general. One one of those is transfer learning. What here we're calling maybe the fine tuning part, where in transfer learning you're taking something that was trained for a certain task and then updating it or fine tuning it to another type of task. And as mentioned on the podcast before, I think that's hugely impactful and a huge benefit for actually people that are doing applied AI. The other thing is this multitask learning framework. I see that this is done in bert. I also see it being done, like I mentioned, in the HMTL model and other cases where this encoder layer is being trained based on being able to do multiple tasks, not just one task. And I would highly recommend looking at that HMTL model as well. This is pretty impressive in that respect.

Sounds good. Anything else?

Well, I was just going to mention, as you can tell, I've kind of been sucked down the rabbit hole of bert, but I did want to mention the people again. This is open source. You can read the article from Google, but also you can go to their GitHub and they have the pre trained models that you can go ahead and use. But there's also actually already been an implementation in Pytorch by Hugging Face and it's not maintained by the Google team but by someone else. And I just thought it was pretty cool and useful to already see that implementation in Pytorch so soon after seeing this stuff come out of Google. So keep that in mind whether you're working on PyTorch or TensorFlow, not that those are the only two, but I think that covers a lot of people. You'll be able to utilize this tech. So that's true.

And I think we end up talking about TensorFlow and PyTorch right now because there's so much coming out in terms of advancements being made where people are really centering around those two platforms. But as you said, there are tons of great tools out there. We're not trying to exclude anyone on those and we would love to hear back. If we are not talking about your favorite tool as much as you'd like to hear, join us in the Slack community and tell us what you're doing in it because we really go out and see what people are writing and talking about and then we end up talking about that on the show. So we definitely would love feedback and whether it's that or other areas, steer us in directions you want to hear from.

Yeah, definitely.

So I noticed that there was another release and this time it was from Facebook. They open sourced their applied reinforcement learning platform. It's called Horizon. And with that I noticed that it's pretty cool. I think if you're not familiar with reinforcement learning, that is an aspect of machine learning where you are using a software agent to take actions that are in the environment that you're operating in. So if you have a model that you're developing and actions are being taken and through those you are trying to reward when things are going the right way and you're trying to learn. So as your model is converging in the right direction, you reward it and you don't reward it when it doesn't. And you see that in a lot of different applications, everything from different AI learning how to play games. You see it a lot in robotics and so it's really great to see Facebook open sourcing, how they're approaching that because they're doing a lot of work on this. Had you seen that one, Daniel?

Yeah, this is definitely interesting to me and I'll Note as well in a previous episode, so episode 14, Wojciech Zaremba talked with us for a whole episode about reinforcement learning. It's an area that I definitely want to get up to speed on. So I did run across this. It was also one of the things that, you know, kind of crossed my path multiple times over the past couple of weeks. One of the interesting things that, that I thought was interesting about this framework that they. Open source or really it's. It's more of a platform, right? This reinforcement learning platform is that it's not kind of, it's not just like a specific library for Pytorch or something. It is actually like a platform that utilizes multiple open source projects to do help you do the task of reinforcement learning. So I see that Spark is involved here along with Pytorch, along with scipy, along with OpenAI Gem and the Onyx framework, which I'm a big fan of and excited about. So you've got the kind of large scale data processing element that's kind of coming from Spark. You've got the scientific computing and numerical machine learning pieces coming from scipy and Pytorch. And then there's other things as well, including model serialization and interoperability that's coming with onnx. So it was really cool to see that this, this kind of convergence of multiple different projects to enable this. You know what seems like a really great platform for actually enabling reinforcement learning and production.

Yeah, I noticed. I was looking across their GitHub page thinking of it as a platform rather than just a library for another platform. You build it in Python using Pytorch for the modeling and the training, and then you can serve models with Caffe 2. So it does have dependencies with other platforms, specifically Pytorch and Caffe 2, but it's a whole system unto itself.

Yeah. Yeah. And I don't know, this was actually pretty surprising to me and maybe this is just my lack of following a lot of reinforcement learning things, but it was kind of a shock to me for them to describe how they are using, how Facebook is using reinforcement learning in production. So they mention on messenger, on 360 video and more. And that was a shock to me. If someone was to ask me before I read this article, you know, who was using reinforcement learning in production, I would probably just kind of give them a blurb about how it's mostly a research thing right now and OpenAI and DeepMind and other people are using it for robots and other things, but it's not really, it hasn't really filtered into production usage and industry. And clearly I'm wrong about that because they're using this, you know, practical platform for reinforcement learning in production. At least on. At least on a few things. They say, you know, Horizon has allowed us to improve the image quality of 360 degree video, optimizing bit rate parameters in real time and other things. So this is actually like, you know, real usage of reinforcement learning rather than just kind of like funny videos of robot arms and stuff. Yeah, this was a pretty big shock to me.

I have seen it used in industry, but it was strictly in robotics. When I was with a previous employer and we had several teams doing some fairly advanced robotics tasks. My team was not. We were very much focused on the computer vision side of things with Mask R, CNN and other convolutional technologies. But yeah, I know another team that we were working with was doing reinforcement learning and deep reinforcement learning, where you're combining reinforcement learning with a deep architecture as well to do that on the robotics side. And that's used a lot on kind of strategy for robotics movement and things. So. But my own personal experience had been very specific to that use case.

Yeah. And I mean, even so, I'm looking at their GitHub page for Horizon right now and it says a platform for applied reinforcement learning or applied rl. And of course that fits right in with what we're passionate about on this show, which is practicality. And this has definitely changed my perception of reinforcement learning outside of kind of the domain of robots like you were talking about, which I have never worked in robots. And so to me, reinforcement learning, like, didn't really come across as something that maybe I would apply directly, at least in the near future, but maybe I need to reevaluate my perceptions there. And actually I'd love to just kind of go through and see. I haven't been through the all of the docs of Horizon, but it looks like that you can install it with Docker. So it would be fun to just kind of spin up Horizon and say, at least say, I've done some reinforcement learning. I feel like I could check that box off of my bucket list at least.

Absolutely. And I want to try to find a use case for both Bert and Horizon from a learning standpoint to dive into them. Because it's kind of funny as we talk about these different things that are happening in the AI community on these fully connected episodes. Episodes it is. You have to really pick and choose what you want to do. But we're seeing so much advancement right now in these areas. So I'm trying to find ways of. Since you don't get to do everything in whatever job you're doing in the AI world, I'm trying to find small projects where we can apply those. So if anyone has ideas, I hope you'll share them in the Slack community or on LinkedIn, the LinkedIn group, because that would be very welcome. Things that are scaled that are affordable for people to dive in and have fun with with.

Yeah, and we'll also, just so you guys know, we always try to include a bunch of links to what we're talking about in our show notes. So there's actually, I have a list here right now of all of the things about Bert. Like I said, there's been a lot. There's been a Google article, TensorFlow, GitHub, there's been a paper on the archive, the PyTorch repo, New York Times article. There's even like a Collab notebook. It's like Jupyter notebook but kind of Google Doc style. So there's one of those for you to try it out. Of course, like I mentioned, Horizon has the Docker install and all of that, so barrier to spinning up a lot of this stuff is a lot lower than it used to be. Which like you mentioned, Chris, in some ways it's, I mean, in a lot of ways it's, it's super exciting, but in other ways it's like there's too much to try.

So there is.

I probably need to focus my attention a little bit. But yeah, so I think that was pretty much what we had to say about Horizon. I'm excited to dig in more. Have you seen anything else in the news recently, Chris, that you want to highlight?

Yeah, I ran across a blog article that's called Does Synthetic Data hold the Secret to Artificial Intelligence? And it caught my eye, kind of dives into just in general about synthetic data and how it's used and in terms of generating enough data to operate on. The reason it really caught my eye is I had some personal experience from my own work having to do with synthetic data. And I also was interviewed a short while back by Thomson Reuters on a series of AI articles that they were posting on that. And I've tweeted if anyone has an interest, I've tweeted about it and stuff and you can find the article. But really talking about using synthetic data going forward to generate larger data sets, how it fits into unsupervised learning for the future. And in my own experience I found a lot of people tend to say, yeah, we'll just synthesize the data, you know, and there's a variety of techniques for that. We found it very hard to do that. And I'm hoping that on, on our, on my own learning curve that me and the people on the teams that I've worked with can, can figure out better, but that can be really challenging. So the article caught my eye because of the, the HOPE forward and I, and as well as everybody does, I would love to be able to say if I want to hit a particular use case and don't have sufficient data, we can go synthesize the data and train it. When we were doing that manually in terms of trying to generate through a number of automated things at a company I used to work at, we found that the data set, we had a small data set that represented the real life problem that we were tackling. And I'm not allowed to disclose what that was, but we also, we didn't have nearly enough to address it. And so we went and tried to synthesize it through a bunch of different techniques and we found that we really had a struggle with getting enough diversity into the data. We could generate the volume, but it was very hard to synthesize the diversity that we needed to where our goal had been. If you take a synthetic data set and compare it side by side with the real, much smaller data set that we already had, that it would be indistinguishable or close to that. So I would love to hear back from listeners and I would love to hear, Daniel, if you have any thoughts on that about how people are approaching synthetic data and some of the different techniques and successes or failures that they've had.

Yeah, maybe just to kind of pause a little bit because I actually, I don't have a lot of experience this whole idea of synthetic data. But you know what I'm thinking when I hear you talk about this is like, you know, hey Chris, like what exactly do you mean by synthetic data? Because isn't data just data? I think you kind of got into that. But maybe you could describe like maybe a little bit more about why there is a need for synthetic data.

That's a great point. So I'm kind of referencing in my brain, my own project, but because of non disclosure issues, I can't address it directly. So I will try. It is oftentimes the case in industry, in the real world when you're trying to tackle a business problem. In the case that we were in, it was to enhance an existing product, and you will say, okay, this is what we need to go solve that problem for training purposes. And, but when you look at the amount of data that you have, you realize that you might need hundreds of thousands of records or millions of records to train against, and you might have, you know, 2,000 or, or less, maybe a few hundred, and that's, that might not be nearly enough to get a high quality model trained for your purposes. So one of the things that people will do is say, are there ways that we can generate our own sense of reality that looks very much like the real thing? So you're generating more data that looks a whole lot like the data that you already have, but you need more volume. And there's a number of different software packages that can help you do that. And we tried some different techniques in this project that we were working on. The challenge that we had there was simply having enough variability, enough diversity in the synthesized data, so that if you were to hold those two data sets up, the synthesized versus the real life one, the real life one was messy. It had all of the little tweaks and diversity as you change things in real life and you get noisy, messy data to train off of, that represents the real world that you're trying to get a model to represent. That's what it is. It's very hard to do, at least in the stuff that we had done, it was very hard to generate synthesized data that didn't look synthesized, that had so much diversity that you would never realized it was generated. The number of different options for various inputs, that kind of thing. As I go forward and I, I'm sure this will come up in the not too distant future where we have to take a synthetic data approach. I'm looking forward to having other people out there say, hey, this is what worked for me, or what didn't work for me.

Yeah, I was just looking as, as you were talking at, you know, models like we're talking about here, which I'm assuming like the models that you were talking about in your use case, but other cases like robotics or natural language, like the BERT model says, it has, you know, like, you know, hundreds of millions of parameters. Right. So to train that many parameters to fit that many parameters takes an enormous amount of data usually, which sometimes you just don't have, don't have access to. I'm glad you brought up this point. It's something that I definitely feel like I need to learn a little bit more about and would be interested to hear from any of our listeners if they have good resources or pointers on that front.

Yeah. I would note the use case that we were generating from, I'll say it was not a convolutional. I've also done it on the convolutional side with more success because you can take the images that you're using in your convolutional neural network and make adjustments. You can change angles, change size, flip.

Them, or some sort of stuff.

Yeah, there's a lot of image manipulation, things you can do to generate more data there. So had great success there. Unfortunately, the use case that I was kind of describing around was not that. So I just wanted to distinguish between the two. I think it's easier with certain types of architectures than other.

Yeah, yeah. Well, on that note, you know, noting that we all have a lot to learn about multiple things, let's go ahead and, you know, move into the part of fully connected where we. Where we highlight a couple of learning resources that have been useful for us or look interesting. The first one that I'm going to point out, which is something that I want to look into a little bit more. More and maybe order the physical copy of is this is a new or almost. I don't know if it's actually out yet, but it's called Grokking Deep Learning. And there's a physical. An ebook from Manning called Grokking Deep Learning. But one of the things that I was looking at was that there's also a kind of companion GitHub repository which itself is kind of helpful even. Maybe even without the book, because it goes through like from the beginning, kind of from scratch. How do we. How do we kind of understand and dig into deep learning? So it goes through, you know, forward propagation and Introduction to Neural networks, gradient descent, generalizing that back propagation, regularization, activation functions, and really kind of starts to pick apart like convolutional layers and word embeddings and other things more from a scratch perspective and trying to get into those things. So I think that this would be a great thing to go through if you're wanting to really kind of understand deep learning and neural networks at a more granular level.

I have the book and have read it and it is very good compared to a lot of books where they don't give you a sufficient understanding. The Grokking part of the title, I think is accurate in that they. Is that the author really tries to explain those. And so having having the examples in the GitHub, which I had not looked at actually is really nice to have in the book. So I know I've read that and enjoyed reading it and thought it was one of the better explanations out there. So definitely concur with that.

Well, I'm glad that that I wasn't making wrong assumptions there.

The I have one that's very specific and I've done other I've talked about this in different articles, but there on Medium I found a Medium post. I probably am going to butcher the name. It's Natalie Jeans J E A N S on Medium. It looks like it's her only article that I see here, but it's the back propagation algorithm demystified and it's another really good explanation. A lot of people as we're getting into the field, you know, this is one of the very first things you learn and actually if you haven't been exposed to back propagation, it can take a while to really understand it and get it. And so I thought this was one of those articles that if you're a newbie into the field and you're trying to understand just how feed forward with bat propagation works, this is another good place to start. She takes you kind of through the initial concepts about, you know, the, the inputs to a node and what it means to have an activation function and and what those are, and kind of describes backpropagation at high level. And then she goes into gradient descent and there's that's a group of different related algorithms, gradient descent that allow you to minimize your error. And she has some good visuals and some great explanation on that. She talks about what those different variants are and then kind of takes you through some examples using Sigmoid, which is not often used in real life these days, but is a is a good training tool that people will use and then actually goes to what people do use in real life, which is back propagating rectified linear units or you might hear it as relu and so did a good job of kind of giving you a good a good stab at understanding that is so I hope people will go see it, we'll put the link in the show notes and that's it for me this week. You have anything else? Daniel?

Nope, that's it. I think those are great and I think it's great that you brought up today how, you know, we just like everyone else, even though they don't always admit it, are always searching through Quora, always searching through stack overflow and GitHub and papers and all of that. If you run across any good ones that we haven't highlighted, let us know on our Slack team. And yeah, it was great discussion today. Thanks for being patient with me, Chris and helping me dig through some of these things.

Yeah, I had a good time. This was a slightly different type of show than anything we've done in terms of just you and me digging in ourselves and digging in not as experts but as many of our listeners just trying to to take it. So I hope it made sense to our listeners and if we get good feedback, I'm looking forward to to talking about specific technologies some more in the future.

Awesome.

Okay, well I hope everybody has a great week and we will talk to you sometime soon. Talk to you later, Daniel.

All right, see you, Chris.

Bye Bye.

All right, thank you for tuning into this episode of Practically I. If you enjoyed this show, do us a favor, go on itunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend. Whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for Changelog is provided by fastly. Learn more@fastly.com and we catch our errors before our users do here at Changelog because of Rollbar. Check them out@rollbar.com changelog and we're hosted on Linode cloud servers. Head to linode.com changelog check them out. Support this show this episode is hosted by Daniel Whitenack and Chris Benson. Editing is done by Tim Smith. The music is by Breakmaster Cylinder and you can find more shows just like this@changelow.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.

I'm Tim Smith and my show Away from Keyboard explores the human side of creative work. You'll hear stories, sometimes deeply personal, about the triumphs and struggles of doing what you love.

I ended up in hospital with burnout. I just kept ignoring the way that it was making me feel and just kept powering through it. And then eventually my body started to give me physical symptoms to say like, hey, you should stop and listen to me.

New episodes premiere every other Wednesday. Find the show@changelog.com afk or wherever you listen to podcasts.