Deep Learning on a WHITEBOARD and Udacity AI Nanodegree Progress! | Learning Intelligence 25

I got my LSTM model training on a GPU, and even that, I'm not sure if I've got it right. It's still taking about 20 minutes per epoch, but I guess there is a lot of data there. And this is still continuing on from last week in the sentiment analysis notebook from the Udacity, uh, AI Nanodegree NLP section. I'm going over— here it is, this class, sentiment analysis, lesson 17— and up to the final step. So eventually Once this model is done, I'm going to be completed notebook, and then I'm moving on to the machine translation project, which is, which is next. But I'll, I'll show you what's going on with it anyway. Let's look in here. NLP machine translation workspace. We'll get onto that in a second. GPU workspaces. So here's the next project, machine translation overview. Let's check this out. So machine translation is a popular topic in research with new papers coming out every year. Over the years of research, different methods have been created like rule-based, statistical, example-based, Machine translation. With all this effort, it's still an unsolved problem. However, neural networks have made a huge leap forward in machine translation. Have you ever used Google Translate? I have before, and if you've ever used it, I've used it probably for a number of years now, and I remember at the start it wasn't that great. But recently, I would say in the last year or so, it's, it's getting very good. So although it's not perfect like human-to-human translation, it is certainly enough. Like when I went to Japan a year, I a year and a bit ago, it was certainly enough to start a conversation with someone or to bridge a little communication gap. So, machine translation is definitely improving and improving more and more. And as you can imagine, it's such a big problem, such a hard problem to solve because there are so many different languages. I mean, I speak English and only 300 million people in the world or something like that. No, I'm getting my numbers wrong. But English is the third most spoken language in the world. So, above that is Chinese and Mandarin. So, even though English seems to be the dominant language for, for business interactions, there's other languages which have many more speakers and other languages which don't have as many. I'll show you how we're going to do this. We're going into a Udacity GPU workspace. So what is this? Well, essentially it's a Jupyter notebook within the Udacity classroom where you can work on your project. Here is the exact project here, machine translation project. We go through here, verify access to the GPU. So this will tell us if we're running on a GPU. I'll go through that another time. And then this is a project here. So essentially all we do is we have time allocated to run a GPU and we click enable. Do you want to enable? So we're not going to enable that now because I need to save my precious time. But how else would you do this if you wanted to run on a GPU? Well, that's where you come into AWS and you can launch a GPU instance. So if we look here, I've got 2 instances running. There's a micro running a failed startup I worked on, and there's a P2xlarge, which is a GPU running in Asia-Pac Southeast, and that's where this code is running. And so once this code's done, the sentiment analysis— sentiment— sentiment analysis notebook, I'll put that on my GitHub so you can check it out. But I want to show you something. I had a delivery while I'm working on this. I think it's— I think it's going to be good for future videos. Let's check it out. What could be in this bad boy? I'll let you guess. You might have to flip the— flip your screen. I'm gonna bring this in. I'm not gonna open it just yet because I've got some study to do, so we're gonna— we're gonna open this tomorrow. If anyone needs a removalist, I'm available. Been working through the Capstone NLP project for Term 2, and essentially it's create an English to French translator, basically a very entry-level version of Google Translate using RNNs. And if you're not sure what RNNs are, it's recurrent neural networks. What Google Translate does is it takes a sequence of English words, for example, and outputs the output sequence in French. And how does it do that? Well, it takes a large database of English words and a large database of already translated French text. For example, you could go to the CNN website or any really, really major public website and it will have often articles of English and the French translations, which human translators have, have done in the past. And so what it does with that database is analyze it billions and billions of times, works out how the words are related to each other, and then uses that information to predict what a future output will be. So let me show you what my simple RNN model was. If we go here, so the first task was to build, of the modeling section, was obviously to, to input or to create an RNN, and it was a simple one. So just a basic RNN model, just to get a baseline data of what the accuracy would be. And we look here, we have— this is made with Keras, by the way— an input layer, a GRU layer. Now a GRU layer is a gated recurrent unit, and I'll put a blog post in the description so you can check out what exactly that is. And then two fully connected layers to finish off. So we have dense, and then for the output layer, again, another dense one here that's time distributed. It's wrapped in a time distributed Keras layer there. And then for the the final model. This is how we create it. We use the model layer to find the inputs, which are back up here, the input layer, and then define the outputs in the output layer. And then of course with Keras, you have to compile it, define the loss, define the optimizer, define the metrics. In this case, I put the learning rate as 0.01. And then we come down here, let's check out what the accuracy is. Validation accuracy, it's printed on two different lines, but just trust me, that's the validation accuracy there, this little point here, 67%. So New Jersey, Est, Parfois, chaud, un, le, lelele, and pad pad pad pad pad pad pad. So what are pads? Well, they're sort of just like padding with zeros on the end because the output sequence was only of a certain length and it didn't have any words to put here. So it's essentially put zeros onto the end. Now if you understand French, tell me if that makes sense. I'm pretty sure it's complete gibberish because the accuracy was only 67%. But then I also completed the embedding layer, which was essentially the exact same as up top, except we've added in this one little layer here, embedding. And what does this layer do? Well, good question. I've actually set that self— that question as a little task for me, what to do next. But I've kind of found out it creates a lookup table that can be trained. From that, I'm thinking out loud, and if you have a better definition of, of what the embedding layer is in Keras, please put a comment below. I'd love to learn about it. And so would other people, of course. My thinking is it creates a table like a matrix sort of thing, and it pins different words on the table, and as it goes through the recurrent neural network, as it keeps going over and over and over again, it works out, okay, how closely related is this word to that word, how closely related is this word to that word, etc., etc., etc., times that by the whole data sequence, and then it uses those relationships to improve on the training. And that just— it keeps going because it's a recurrent layer, it just keeps looping over itself. And now that was probably a very bad explanation, But if you have one better, please let me know in the comments below. And then we look here, 89%. So it increased it by 20% by just adding that embedding layer. So what does this say? New Jersey est parfois calme en Corse et one automate— no, I've got automate on the brain. Autonomie est est en avril. I'd never succeed well in France. Well, that's enough coding for today in terms of the project. I may go on to the third model implementation in a little bit, but what we're going to do is we're going to open this bad boy. We're going to test it out. I messed up. This bit, this little groove here, should be on the inside. And the exact same thing happened on the other one. Now this is what you get when you rush through building something. I was never the best at building anything from IKEA. I'll show you. This little bit on this should slot into here, but it's no help if it's on the outside. Attempt 2. Ta-da! It's complete. How beautiful is it? Look at it. Look at all this space. I can reach. It's nearly, nearly as wide as me. Check that out. Whiteboard nearly as big as me. I love it. Now, it's a much better upgrade from the smaller one over there. I know that's gonna still be like my day-to-day stuff, and this is gonna be for practicing code. I know you probably can't see it from way back there, and I really wish you could so I could write this live. And yeah, that's not a very good angle there, is it? You can kind of see it, but you'd have to, have to really look in. And now I'm going to run over this really quickly and tell me what you think it is. If you can guess before I tell you and put it in the comments, 100 points to you. But what else would I be putting on here other than a full-blown— this is an end-to-end deep learning network. This is what you can do in a whiteboard. This is what, maybe 20 lines of code if you don't include the, the dependencies up here. I'll get onto this in a second. That's probably 15 lines of code right there. That's a deep learning model to analyze an IMDb dataset. And now what is this? So look, let's look here. This is Keras, by the way. So if you're not familiar with Keras or Python, I'll try to run you through as best I can, but I will put a full link of this in the description into an actual Jupyter Notebook or something like that. So you can, you can play around with this. If you want. And so from Keras, this is the dependencies. So this is how we're going to get the information to start our model. So keras.datasets import imdb, so it's the IMDb database. From keras.models import Sequential, that's the type of deep learning model we'll be using for this network. From keras.layers, we want to import the Dense layer and LSTM layer, because LSTM is what we've been working with. It is also one of the best for sequence models, which is IMDb is the database, is a whole bunch of movie review comments, and what we're going to try and do is identify the sentiment of those movie comments. What's sentiment? Because it's just a funny word for, uh, is the comment good or is the comment bad? That's what we're doing with this network. This, this, this code here will work it out, and we'll see how accurate it is at the end. Then we're importing the dense layers here, and then from keras.layers.embeddings This is separate to, to these two, like these two up here, because embeddings is a bit deeper in the Keras library. We're going to import embedding. That's another layer. Now we're going to set up a variable here, max_words, 5,000. So that means the max amount of words we're going to use, uh, from, from the database is 5,000. And then we go here, x_train, y_train. We're going to set up our, our data here. So training data, test data. X_train, y_train, x_test, y_test equals imdb.load_data. This is built into Keras. num_words equals max_words. So we're feeding the number of words which you defined up in here. Maybe that should be 500. I'm not entirely sure. It'll be correct in the Jupyter notebook, whichever one you have. And so we're going x_train. We're going to pad these, the training and test data. And what does padding mean? Well, we're going to make sure they're all the same length. So if you have one review where the guy has decided, or the person has decided to put their heart out onto IMDb and put it into hundreds of words or something like that, whereas all of you have got is, maybe I've been lazy and I just wanted to go on there and I just left great movie. So we're gonna make sure they're all the same length. And how do we do that? Well, we're gonna use pad sequences from Carers, which is, it should have been up in here. We should have imported that, but that's, that's all right. It'll be correct in the Jupyter notebook. But pad_sequences is a function from, from Keras again, and we're going to make sure that the max_length is the max_review_length, ah, which we've defined here. Sorry, max_review_length. I missed that one. So the max_review_length is 500 words and the padding is after it. So they're all going to be every single sequence in X_train and X_test is going to be 500 words long. And if they aren't 500 words long already, they're going to have zeros at the end. So say, for example, they're 300 words long, they're going to add 200 zeros to the end. Now we're going to get through this and then we're going to create the model. So we've defined our data, our test data. Now we're going to put it all into a model so we can start to understand it. So define the model. First up, it's going to be a sequential model. Don't worry too much about this. It just means that we're going to add layers continuously. So then model.add, add a layer. Add embedding layer with— we're feeding embedding layer max words, which again, 500. Come down into here, then we have embedding vector length, which is— we had here 32. Now what's this? We defined this before, so embedding vector length is 32. So that means we're creating an embedding library of 32 vector— so that means we're creating an embedding library and we're going to represent each word with 32-length vectors. What does that mean? Well, we're going to turn each word into its own series of 32 numbers. That's how I understand the embedding library. And if you have a better definition than that, please leave a comment below and help us out. Embedding vector length and the input length into the embedding layer is max review length, which is 500. Now we've got a few more layers here. model.add LSTM. We're putting an LSTM layer that comes from back up here. LSTM layer, it's going to have 100 units. Dropout going to be 0.2. It's going to have recurrent dropout, which is 0.2, which means because LSTM is like a tap, right? It keeps going. It lets, it lets certain data in and it lets certain data go through. So we want to have a dropout applied to the end of it. And recurrent dropout is applied to the data that's fed through the LSTM cell. Dropout on the end and dropout within the actual LSTM cell. That's what those two parameters are. And then we have model.addDense, which is a fully connected layer because we want to fully connect all of our LSTM units here. And then we're going to have only one output for this one because— why do we want the only one output? Because it only has to be positive or negative. That's the end goal of this model. It's a sequence-to-sequence model where the goal is to identify if the movie review is positive or negative. And so we have here activation sigmoid, which is best for 0 or 1. Then we compile the model. In Keras, you compile the model always before you fit it. So compile, we define our loss function, which is binary cross entropy. What's binary cross entropy? Well, just imagine binary is 0 or 1, right? So we want to work out positive or negative, 0 or 1. And then we define the optimizer, which is Adam. And then we define what metrics we want Keras to list, which is accuracy. Now that's built in. So when you run it, you'll see a bunch of numbers print out saying how accurate your model is. And then we go here, print_model_summary. That will print a summary of your model so you can check it out layer by layer. Essentially, it will say, okay, we're starting with the embedding layer, we're going to the LSTM layer, and then we're going to the dense layer, and it will, will tell you about that. Then to fit the model, to, to have it fit the data to predict what's going on, we have model.fit. We feed it the training data, so X_train, which is our X labels, Y_train which is what we defined before up there, and then number of epochs, so 3 times. So that means it's going to go over, loop over the data at least 3 times. And then we come back in here and batch size is 64. Now what does batch size mean? Well, batch size means it's going to go through 64 points of data at a time. So rather than go through one, one review at a time, it might go through 64 at a time. So just imagine it like that. That's what batch size does. And then if we go here, Finally, we're going to evaluate the model. So model_scores, our last variable, model_scores equals model.evaluate. Again, this is, this is inbuilt in Keras, and we're going to feed it our test data because why? Our training set is separate from our test set. And now if we fed it our training set in here, it might get different results. So we want to make sure that if we're evaluating our model, it's on the test data, not on the training data. And verbose equals 0 just Ignore that for the time being. It just means what is it going to print out. You'll see what's in the Jupyter Notebook as well. And then so finally, we're going to print accuracy equals some, some funny symbols here, but essentially it means print something, print it with two, two dot points at the end. That's what that, that little thing means there. And we're going to feed it scores 1 times 100 to get it percentage. And that's the output there, 82.82%. And that took, that took 10 minutes to find that model, but that's all right. That's what we did with just, just that code there. Look at that. Just that, just that, we were able to get 82.82% accuracy predicting whether a review from an IMDb, from a comment, you could, you could probably extrapolate this out onto a bigger dataset. We've only used a small one here and we've managed to get 82.82% predicting whether the comment was positive or negative. I think that That is incredible. And that is in such a short, simple model. So imagine what you can do when you expand this out. Just a little heads up, if you want to read my comparison of the Udada— Udada. Just a little heads up, guys, if you want to check out, I wrote a fairly lengthy blog post. There's a short answer, short answer, and a longer answer. There we go. On Quora, as an answer to the question Udacity's artificial intelligence Nanodegree program versus Andrew Ng's deeplearning.ai course on Coursera. I'll let you read the full thing. I'll link it in the description. Essentially, what it is, is describing my experience and the differences between the deeplearning.ai course and the Artificial Intelligence Nanodegree and the Deep Learning Nanodegree. It'll be on Quora and eventually Medium and whatnot. So, the links to find it all will be in the description. Otherwise, been working through the Artificial Intelligence Nanodegree NLP machine translation project. And you know what? I'm almost finished. But you know the saying, have you heard the saying where it's like once you're 90% done on something, you've still got 90% to go? That's where I feel I'm at. And so I'm gonna give it a break for the rest of today because I've sort of hit a coding roadblock where I'm just looking at the same problem over and over and over and over again. Been working through it all day. So I'll show you what we've done. I've just built a custom model, an RNN model, using everything I've learned from the previous models that incorporates embedding and a bi-directional RNN into one model. So here we go. I've built an RNN with embedding, encoder-decoder, and bi-directional layers. So the input layer we have here, the embedding layer, encoder with a GRU, which is a gated recurrent unit, which is similar to an LSTM, a bit less complicated, repeat vector for the encoder-decoder. Then we have the decoder, which is— this is the bi-directional layer. This may not be the best architecture. I'm still running through that empirically to test my results. And then of course to finish off we have a dense layer which is time distributed, which takes, takes a ReLU activation. And the final layer is another dense layer with a softmax activation. Then we compile it here like traditional Keras models. What we're going to do is we're going to use it to predict some French text. And where do we have— what is it? So he saw an old yellow truck. Let's compare this to Google Translate, hey? Where's my output? There we go. Il a vu un petit camion jaune. Let's see what Google says. Il a vu un camion jaune. My model has some improving to do, and we'll see if we can do that by the end, by the time I submit it. However, the accuracy of mine is about 95%. So, we've got some, we've got some work to do. I mean, I stuffed up here. I'm trying to do the optional extra or the optional enhancements, which is split the data into training and test sets using sklearn.model_selection.train_test_split. When I did that, it completely stopped improving the accuracy. So here's what I've got to do next. See here, fixed model accuracy not improving for all models. And so a tip that I always do is set yourself before you finish the day or before you finish any, any major thing, or when you take a break, a big break from something. And for me, a big break is overnight because by tomorrow you've probably forgot where your mindset is at. I leave a little checklist of the next steps I can take in something, whether it's what I have to do tomorrow, and I try to do this every night at least before bed, and I'll write down maybe 6 things or so that I want to get done the next day. So, I've got my work cut out for me over the next couple of days. Woohoo! What say we wrap up this extra long video? Yes, Daniel, that would be a great idea. Okay, let's do it. So, last week's question of the week was, I asked, what is your favourite YouTube— no, what is your favourite learning platform? And the winner of it is Moto G5. Thank you so much for your comment, Moto. And your answer said YouTube is your favourite learning platform. And thank you so much. I mean, I'm on YouTube, Suraj is on YouTube. There's so much to do on YouTube. It is, it is amazing. I've learned, I've been watching Peter McKinnon this week. He does some great videos on how to improve your videos. So, that's what I'm trying to do. It's just an amazing place to learn. And otherwise, you also enjoy university. Me too, I learnt a lot at university, but at the moment I think I prefer YouTube and online. This week's question of the week: What are you working on? Is there a project that you're having a great time with? Are you excited about something? Are you learning something on YouTube? Are you learning something on Coursera? Are you doing a Udacity Nanodegree? Let us know, what are you working on? Best comment will get a shoutout in next week's video. How did you like that coding session I did in this video on the whiteboard? Over there. Oh, you can't see it. It's not on there at the moment. But how did you like it? If you want to see more of that, leave a like, leave a comment. I'd appreciate your advice or your feedback. Should I do more of those? Uh, whiteboard's probably not the best thing to do it on. Uh, maybe, maybe we do some Jupiter Notebooks in the future. Don't forget, check out the Jupiter Notebook down below to, to see what was on the whiteboard. Next week we're going to finish off the NLP capstone project. Going to have it submitted. Hopefully, fingers crossed, we, we pass on the first go. If not, We'll just, we'll just keep hammering until we get it done. But as always, thank you for watching and— no, I did that last week. Keep learning.