What Is ChatGPT Doing? Episode 5: Inside ChatGPT

Hey, everyone, we're back again, John and Julio.

Hi, John.

Questions? Hi, Julio. Julio providing answers. I think last time we learned a lot about neural networks. And this time in episode five, we're going to do more inside ChatGPT, which conjures up all sorts of images. So, Julio, I'm ready to get started. How are you doing? You ready?

I think I am. I think I am.

So we're now on episode five and we're going really inside chatgpt. Last time we talked a lot about neural networks and that seemed to really focus on numbers. But I think now we're going more into all sorts of input strings, images, all sorts of things. Can you kind of explain how we can go from dealing with numbers to now dealing with images and words and stuff?

Yeah, we saw last time how you can kind of train to predict a number from a number. And actually that's always going to be the case. Like these models will always be trained on numbers. So you need some kind of connection from your input and your output to the numeric representation. One of the easiest thing to do to visualize this, you mentioned images. And there are like many networks that can work on images. So for example, this one is a famous model which is trained to basically read in a squared image. You see this one, 2828 is a grayscale image of a digit. And at the end it predicts these list of 10 numbers. That is the probability of being a digit between 0 and 9. And we can see that if we just evaluate this on an image, it becomes a four. So what is happening here? In this case, we do the trivial thing. We just take the image representation as a matrix of colors. Matrix of colors has numbers, rgb, in this case, just gray, black and white. And that's it. That is our numerical presentation. And that representation is processed through the network.

Gotcha. I have to comment on the irony of starting with images. And the images are of numbers. So just.

Well, you got me there.

So how do we make sure that we represent all of these different kind of inputs, like just the strings and like you've got all of these different sort of things. How do they come out being like normalized as these vectors? How does that kind of.

Okay, so let me add an intermediate step here. We see that this is kind of a deeper network than what we, what you had before. And you can imagine that the initial part is kind of processing the numbers that compose the image into a vector representation. And then just the final part here is reading these 500 elements vector representation and Turning it into. Into the proper list of probabilities. So for example, if I evaluate here just the network minus the last three bits here. So I just only get this 500 elements. I see that this, I can turn this four into a vector. I remove. The short is a long vector of numbers. And these numbers are somewhat correlated for images of the same digit. So you can see like all the fours here, they have a numerical representation which is kind of similar in this vector space. And all these sixes and all these eights. And this thing can be leveraged on its own to do some sort of unsupervised learning, let's call it. So if I just for example, try to compress the 3D, this long vector of 500 elements, I can try to do a representation of these digits in the 3D space. So in this case I start from an arbitrary number or can be an arbitrary thing. And when I have it's embedding, I can. For example, you see all the ones are here, some zeros are here, fives, seven threes.

Gotcha. So one of the big takeaways is that once we represent the image string or even number as a vector, then it's just the same process kind of after that. Is that right?

Yeah, that will depend a bit on your. Like what you say last time, you could have an arbitrary large network with all possible connections. But that's very expensive. So for images you tend to have networks that looks like a filter, or you used to. Now we are using transformers for everything. Transformers, they look at like bits and the relationship between the bits. For example, for GPT, it looks at the sequence and it grows with the sequence. Do you want me to show you a bit more how this system works in the GPT environment?

Absolutely. The closer to GPT we can get, I think more interesting. Not only me, I will be, but then also our viewers.

Okay, so let's use GPT2, which is public. We can actually look at the model itself. So this is the beginning of the network and is the embedding module. So we see that it gets an input which is a sequence. Because I'm going to pass my whole sentence. And the sequence is split in different parts. And these parts are called tokens. So I can actually show you what these tokens are. So like for example, let's take this simple string. A large gray tabby cat was lying on a rocking chair. And if I extract what the inputs processing in. You see, we talk about this. The whole sentence gets split into a bunch of integers. And this is the first thing, and this is used to get to a basically internal vocabulary which is not of words but of numbers. And what are these numbers? If we extract also the vocabulary, then we can look back and say, okay, so these are. See, this number 62 was like a large gray tabby. It was not in the vocabulary, so it got split in two. And this is a typical thing of this type of tokenization procedure. This is very flexible. So if it has the word, you just represent it with a single number. If it doesn't, you have to split it in sub parts of words. It's kind of a nice flexibility.

That's kind of the difference between tokens and words. Is that kind of right?

Yes, correct.

What you're saying in like, with the vocabulary, if I understand is that it may have the complete word and then it gets represented by a number. If it doesn't understand the complete word, it just splits it and then represents whatever it finds.

Yes, correct. So let's. If I take this thing, let's. Let's give it a name. Tokenizer. Yep. And I can apply it to a word, which is. Hello. So in this case, like, there's one trick. I put a space here because, like, most of the words, they come after something else, so they are tokenized together with a space. So if I do this, you see the number changes. So, like, this is a single thing, but if I put something longer, I don't know, like serendipity. Yeah, serendipity is not a single token. Is it spelled correctly? I don't know. Whoa. Okay. Yes. So this is what it was represented with.

So I think we were talking about. We've talked about the tokens kind of last time. So it's good to hear more about tokens. What is the direction kind of of the model as it does this tokenization and then goes on to the next step. Can you tell us more about how that works?

Oh, yeah. I got a bit distracted with the token. So we were looking here at this part with. We were talking about embeddings of images, embedding of text. So the tokenization part is important because you're not directly computing the embedding of words or like a sentence. You are first doing the split, and then this becomes a sequence that is n times 768. This is basically a sequence of tokens, individual tokens. So these tokens are embedded in a larger vector space with the operation that I'm visualizing like this. So this is what these two modules are doing, this embedding, and this Positional embedding. And you see here I put a sentence which is very simple. It's a bunch of hellos and a bunch of bys. And you see that now each hello is represented by a single vector. Each column here is one hello. And they are all the same because it's always hello. And each by is another type of vector, always the same, but they come at a different position in the sentence. So this hello is position one, this thing here is position two, position three, and so on and so forth. So they also get to distinguish between them. Down the line of processing, they get another part which is a vector that depends on where they are, and these two vectors. So this one is created from the integer that is the token, and this one is created from the integer that is the position. And it's just done by a linear operation. So just a big matrix that is like distributing them in a very big vector space of these 768 dimensions. So in the end, what the network does, it sees the whole sentence as something like this. These metrics of vector representation for each token.

Gotcha. So it's like in many other ways, in many other linear kind of operations, it just adds the two results.

Yes, this is. I think people tried different things like in the original transformer paper, I believe they were catenating the representations, just making a longer vector. Here they are summing it in the end is mostly what it works. So you. You try to do something sensible that preserves the information and then you see if it works better one way or the other.

So we had this embedding kind of process. How do we combine these elements that then have some final output that we can use?

We do extra processing on this. This embedding here is just the beginning. Oh, this is. Yeah, this is. This is very small. Yeah. Not a billion parameters model. Like, this is just the beginning. What we do after this is we use the famous attention. Yeah. So this thing here that I'm showing you, and we're going to unpack it a bit in a second. This is the module that is taking as input this embedding and is picking it out a new embedding, which is a transformation of the input one. And the thing that the input and the outputs are the same. It means that I can stack these modules. I can have like six of them, 10 of them, 20 of them. As much as I can handle, let's say. So what happens here? You see that there are two parts and there are like two plus signs here. This means that these are two operations that Are adding a perturbation on top of the original input? Yeah, I'm not producing something directly. I'm producing something which is sum to the original input. So I'm producing a perturbation of the original input. This is the standard technique to kind of make it a bit easier to reach a result because you expect the result to be a bit closer to the input than just to learn from scratch to go to some place. And this initial part with the I here is our presentation for the attention. And we talk about this paper. Attention is all you need. And this is actually the procedure. So this N is just some normalization to keep everything in check, the same ballpark. And then here you produce three vectors starting from the original representation that are called key, query and value. And it's sort of a database system. One acts as a question, one acts as a placeholder, the key and then the value acts as what you actually want to recover. And you see they all get fed inside the attention. So basically this means that for each element of the sequence to compute the new value, I use itself and all the others as a question and say given that I have this sequence, how should I recombine these vector to get the new representation for each element? And this is what the attention is computing this big matrix which is saying, oh, if you have this and these, you should recombine these two this way to get the new result and then the diseases packaged back in.

Okay, so you may have covered this before, but then why do they name, you know this I attention.

The core concept is that once you have a sequence, let's say like this. And. And I want to predict like I don't have this part, for example, I want to predict an extra term. Should I just look at. Should I just look at rocking? Should I just look at was what is more important? So while we are processing the vector representation of these elements, we are saying the new one. The new representation for rocking should take into account the fact that for example that there was lie that there was cat. Maybe not so much large, but this is all free form, this network. This model is free to kind of build up its own representation of relationship between words. And basically these matrices that create key, query and value, they encode these representations that they get then recombined. So it's kind of a way to give some leg room to learn relationship between words in a way that is not strictly imposed by us.

Gotcha. So it's kind of like the same way I'm going out on a limb here, might be Dangerous. Might be dangerous for the cat.

Oh, poor cat.

So it's almost like it's kind of looking at all of the words and then saying like here's kind of this relationship, these features that I have to take that and then decide what that next thing might be. That next word.

Yeah, yeah, exactly. At the end, at the end of it, at this point, this is just reshuffling all these representations together multiple times.

And again, like the end result is trying to have this way of, for the neural network, this way of understanding words, like the relationship between words.

Yes, I would say so, yes. And at the end you need to go from a vector like this. You know what you do is like if we take again this sentence, your final task would be that you start from a sentence of length one, like a, and you want to predict large. So you process this vector until the end and then you use again a linear projection on this 50,000 tokens vocabulary. And you say what is the probability for each one of them? And then the correct result here, here we're looking at training, not a generation. So the correct result is large. So this is my correct distribution is zero everywhere and one where large is. And then you can feed it back, you can say a large, these two. And then using the final representation, you want to predict gray and then a large gray. And you want to predict tab, tabby, tabiquette, a large gray, tabiquette was, and so on and so forth. So you can see how it's easy. Like any sentence can be converted into training set for this model.

So in this case we have like just one sentence as kind of the, the training data. But Chat GTP has like, has been trained on like lots and lots of data. Does that get delivered as like sentences or like pages of data? Like how does that part, how does this relate to Chat gtp?

Okay, so this is only the GPT part that is trained like this. And there are two answers to your question. So the first answer is you can connect all the sentences together. It doesn't matter if they come from different sources or not, as long as you add something like this, for example. So at the end, after the period, I want to predict that this is over. So this is teaching the model that sentences, they do end, otherwise you would keep predicting forever. But you want the model to kind of understand that there's a unit of a unit of reasoning where at the end you stop.

Right?

So yeah, this is like how, I don't know how actually OpenAI was training it concretely, but this is the way you can train one one of those. So you just catenate all these sentences and then you just put an end token.

Okay, so we're just talking about chat GPT and the difference in training between that and just plain old regular no chat. So can you talk more about, like that training process and how that works?

Yes, let's call what we've seen so far, which predicts this, like in a completely unsupervised way, learns to continue these sentences. Let's call this step zero. So we don't know exactly the details, but from the OpenAI blog about Instruct GPT, which is a precursor model of ChatGPT, we have this nice image that kind of explains what the process was. So if GPT is step zero, step one is that you don't just continue arbitrary sentences from the training set, but you actually have a curated list of sentences with a curated list of continuation, which are written by humans. So this was specifically created for this training. And then you do the same operation of training for the continuation, but in a more targeted way. It's not just data in the wild, if it makes sense. Okay, so this is step one. Then we have a slightly better model that instead of just learning random sentences, it has learned, for example, question, answer, or prompt and continuation more in general. So we take this model and. And we need to make sure that it does what we want in a very reliable way. So we take the model, we take the same database of prompts, and we use it to generate multiple alternatives. And then we have a human being scoring the alternatives. Why scoring? I mean, scoring with preferences, not with numbers. Why this step is important because if you ask me what's my favorite food, I may say pizza is 10, and I don't know, burger is 6, but you could say, no, pizza is 8 and burger is 7. We agree that pizza is better than burger, and I'm sorry if you don't think that way, but we don't agree on the actual mark, you know, the final score. So it's much more stable if you just ask people to say B is better than A, which is better than D, which is better than C. So this is what they did. They asked a lot of people to rate these continuations, and then actually they trained another neural network, like a support neural network, to go from these preferences to an actual number to an actual final score. So there was only one judge that was taking the preferences and converting them to numbers. And this is what is called reward model. So this is step two. Then you have your refined model from step one, and then a reward model that was trained to kind of understand human preferences. And then you go to step three. You put these two models in a game, and model one generates continuations. And then model two, the reward model, say, okay, this is good, this is bad. It gives an actual reward. And this is slightly different. This is not standard training because you're not giving a score for each and every token. You are just waiting for the model to train the whole sentence. And you're saying, like, this is a four. Just try, try again and say, okay, this is very good. This is an 8. I believe you do differently in the US you go a to F. So in this case, I believe I use numbers. So in this case, you can loop this and the reward model will kind of keep teaching what is becoming the ChatGPT model here to get better and better.

Gotcha. So again, this kind of slices just the plain GPT versus, like, the chat GPT that we all, like, know and love and interact with. Is that kind of right?

Yeah, maybe they do more. But this is kind of a. According to the blog is the gist of it. Like, there's a bit more than just randomly repeating what's written on the web. That's it.

Gotcha. Cool. Wow. So it seems like we've covered a lot, like going from neural nets to then getting to more of how chat GPT actually functions and trains and everything else, and kind of reviewed a lot of the concepts on the way. Absolutely. Thank you again, Julio, for going through different things and putting up with my questions. I really appreciate it.

Thank you, John.