Wolfram Student Podcast S2 Ep 9: Analyzing GPT-2's Brain Development

Hello, everyone, and welcome to season two, episode nine of the Wolfram Student Podcast. I'm Sam. And I'm Reshank, and we're your hosts.

This is the Wolfram Student Podcast, where every fortnight we dive into a new, innovative project done by high schoolers using the Wolfram language.

For this episode, let's welcome Shreya, who will be discussing her Wolfram Summer Research Program project on analyzing GPT2's brain development. Hi, Shreya.

Hi, I'm Shreya Ramanan. I'm currently a junior at the Dulles High School Math and Science Academy, and I'm from Sugarland, Texas, and I attended the Wolfram High School Summer Research Program.

Yeah. Cool. Can you give us a quick summary, you know, of what camp was like this year?

So camp was a really interesting. It was a really fun experience, in my opinion, because I hadn't really worked with the Wolfram language before this, and so this was basically my first time working with it. So. And how camp was, like, was we got to do, like, one on one mentorship with Stephen Wolfram, and he basically assigned us our topic, and it, like, basically fit into our interests. So my interests were mainly centered around AI and LLMs and how those work. And then we got some time to, like, get comfortable with the Wolfram language and then start making our own project. And by the end of camp, I felt pretty, like, a lot more confident with the Wolfram language because even though I hadn't used it at all before, I was able to come up with, like, a complete research project with the help of, like, mentors and friends. And it was, like, a really interesting way to delve into one of my favorite topics.

Yeah, that's great to hear. I'm glad you had a great time there. And just one last question we like to ask everyone who comes on. What's your favorite Wolfram function?

My favorite Wolfram function, it would probably be replace part. It's kind of like an odd one. But also I think it's really practical and useful because you don't have to actually. It has a lot of different functions, and you can do a lot of different things with it. So you can go back into really long lists or lists of data and, like, change small things that you and, like, test out how that affects the entire code. And so I think that's a really interesting function.

Yeah, that's really cool. So before we dive into your project, can you first give us a quick summary of what it's about?

So my project is basically looking at the brain of the GPT2 model. And so basically, GPT2 is this LLM. It's this like AI thing that, that you enter a prompt and it'll come up with its own response. So it's really similar to ChatGPT. And what I'm doing is I'm basically altering various factors of the model, such as the temperature or the nodes, things that play a role in delivering the output to the user to actually understand how the model works. And I'm trying to see how, like, the different parts of the model, how changing those can explain why AI hallucinations occur as well. And so that's when these LLMs generate incorrect information or start generating nonsensical outputs.

Yeah, that sounds really cool. Let's just go right in.

So my first thought was how the different parts of the model that I can alter. And so I had to do a lot of research with this and ask my mentors as well. And so the main things that I'll be going through is basically altering the temperature, altering the nodes, and altering the different weights within those nodes. And I'll explain what those are later on. So first what I did was I had to set up the GPT2 model and I actually had again, API key for this, but basically importing the model into the Wolfram Mathematica notebook and I'm declaring it as the variable GPT2 model so that I can reference it later on. And when you click on this, you're able to see that kind of the different information about the model itself. So I'm able to see that it is a transformer structure. And so that means that it takes a little more time to train and it's a little slower compared to other models, but it's also one of the simpler AI models compared to others. And so in order to actually make it generate text based on an input, I had to create the generate sample function. And so what I'm doing is I'm listing the parameters. So the language model name, which is the GPT2 model that I defined earlier, the input string, which is my input into the ln the number of tokens. So that's basically the length of the response. So each token, it's kind of like a unit of text, so it's about three quarters of a word, and then the temperature, which affects how creative or like how interesting kind of the output is, if that makes sense. And basically what I'm doing is the model is taking a random sample based on the input that I give it. And so it has like a bunch of data that it's being trained on. So it'll take like one of the random samples from the data and it based on the temperature that I input, and then it'll join all of those samples together to create one single input based on the number of tokens that I said before. And so basically now that when I'm calling the function generatesample and I'm listing the name, I'm listing the input and the number of tokens, so that determines the length. So it's around one to two sentences. For 30 tokens, it's able to give a response that generally makes sense. And it's pretty like it make it makes sense and it sort of relates to the topic, but not completely because the GPT2 model isn't quite accurate compared to later versions such as like GPT3 or ChatGPT. And so now what I'm going to do is I want to look at the information within the model. So I'm trying to see like the number, like all of the layers within the model and how many there are and the weights that each layer holds. So basically the layers each have a different role in the output of the LLM in terms of processing it, training the model and extracting information and generating and outputting a response. And so basically each layer has a bunch of neurons, which are basic units that process the information. And the weights within each control the strength of the layer by determining how much influence one neuron within the layer has on another. So more weights means that it has more of an influence on the GPT2 models output. So basically what I'm doing is I'm getting the information about the arrays and compiling it into a list and seeing how many the length of that list, so how many arrays there are. And then I'm calling out a specific array and asking it to show the weights within it. And so basically it shows that there are 196 arrays and each one contains around 700 to 800 weights. And then you're able to see a list of all of the weights. This is specifically for array two. And so that shows how like, each weight is completely random, but zeroing out all of the weights within a specific array can have a significant impact on the model because that's like zeroing out 700 to 800 weights. And so that might affect its function. And so the first thing that I'm trying to do is seeing the effect of altering different tokens. So when I'm inputting the like a sample text into the GPT2 model when I'm calling that function, basically what it's doing is when it's processing the text that I inputted, it's also generating text token by token. So when I'm zeroing out different tokens in the response, it's like interrupting the GPT2 models train of thought. And so that might affect the output and its coherence. And I'm seeing whether I change certain tokens towards the beginning or end of the response, how can that affect its output? So what I did was first I created this variable called generatedtokens, and I'm storing all of the tokens that the model creates into this variable, and then I'm defining the start and the end number of tokens that I'm going to zero out. So in this example I'm zeroing out tokens 1 through 10. And then after that what I'm doing is I need a function that can sort through the tokens and replace tokens 1 through 10 with 0. So what I'm doing is I'm using that replace part function that I used earlier, and I'm basically going like sorting through each of the tokens, and from 1 to 10, I'm replacing that number with 0. And after that I'm able to input it into the. I'm able to use that generate sample function. And I'm basically using the modified tokens and generating an output. And so what I'm doing is when tokens 1 through 10 are zeroed out, I found that the response doesn't really make sense. But I need to find a way to actually quantify this so that I can compare it to other outputs. So two functions that I use to actually quantify the coherence of its outputs are the text structure and dictionary word queue. And so those two, the text structure basically lists the, like, different parts of speech. And I'm analyzing, like, whether there's proper punctuation, like whether the sentence is grammatically correct, stuff like that. And then for the dictionary word queue, the more number of words that are in the English dictionary, which is what that function does, that's better because it shows that the AI model isn't just making up random words, because that would show that it's kind of like not really creating outputs that make sense. And so that would show how it's more incoherent. And so when I'm using the same sentence that was found when I zeroed out tokens 1 through 10. And so when I'm looking at that, I'm able to see that a Lot of the punctuation really isn't in the same, like in the right places and they're like random symbols. And a lot of the words also don't even make sense because when I'm counting the number of dictionary words that aren't in the, the number of words that aren't in the English dictionary, I'm able to find that there are two words that don't, that don't exist in the first place. And I'm also going to repeat that same process with tokens 50 through 60. So I use that same function, the modified tokens function and the replace part that I did before. I'm zeroing out tokens 50 through 60 with zero, and then I'm going to generate a sample that has 70 tokens. So as you can see, if I'm zeroing out the two like ten tokens closer to the end and seeing how that can affect its output if I'm zeroing out tokens at the end compared to at the beginning like I did before. So over here I'm generating a sample with the same input text that I did for the previous one. And I'm able to see that the output like, is a lot more coherent. It's in full sentences. There aren't really many words that like, don't exist at all. And so I'm going to repeat that same thing that I did where I do the text structure and the dictionary words. And so all of the, so according to the text structure, all of the like, proper word forms are in the right places. And like there isn't random punctuation scattered anywhere like the previous one. And then there are zero words that are not found in the English dictionary, which is good because that means that the AI model is coming up with outputs that actually make sense and aren't just made up words. And so that's a really drastic difference from the one before because it allowed me to see that when you zero out tokens towards the end of the model's output, compared to at the beginning, the output is a lot more coherent and it's not just like random words kind of jumbled together. And then the last thing I did for this section was I zeroed out all the tokens just to see how that would impact the model's output. And so over here I'm storing all of the tokens that the model generates into this variable generated tokens, and then I'm sorting through all of it and setting each token to zero and inputting that into a variable called modifytokens. And then I'm using that generate sample function that I did for the previous two ones and I'm using the modified tokens for that. And then once. And then I use two sentences for like, for the output when all tokens are zeroed out from the model. And so for one of them, as you can see, the output doesn't really make sense. And there's random punctuation, there's like brackets, those curly braces, and those aren't like placed in the right ways. And there are a lot of words that aren't like real words or they don't relate to the prompt at all compared to when I did token 0 through 50 through 60. And so what I did was I used the same functions as before, so the text structure and the number of words not found in the English dictionary, and it was able to count that there are three words not found in the English dictionary, which just. And that continues to show that when either all of the tokens or the first few are zeroed out, that has a lot more significant impact compared to when tokens later on are zeroed out. Because when you're zeroing out tokens towards the beginning, like zeroing out all of the tokens, you're essentially still zeroing out tokens 0 through 10 or the beginning one. So that greatly impacts its output. And then I basically compare that with when I'm not zeroing out any tokens. And as you can see, the punctuation is placed in the right place. It also relates a lot more to the prompt that I entered. So the prompt that I entered was AI is cool because. And as you can see, like the sample is a lot more like technology related or it like relates to like benefits of AI, stuff like that. And it doesn't. It only has a few minor errors when it comes to the sentence structure as well. And so the next thing I did for my project was I zeroed out nodes. So as I mentioned before, nodes are basically like layers and each layer plays a specific role in the LLM's output when it comes to processing the output and outputting a response, or processing the input and outputting response. And then each node has a specific weight and it's like a list of weights that control the strength of each layer. And so what I wanted to do is zero out all of the weights and see if like the extent to which the weights have an impact on the GPT2 model's response. And so what I did here was basically I displayed all of the arrays and input it into this Variable called list. And then I'm sorting through, sorry, I'm sorting through all of the arrays in that list and I'm sending it out to zero. And so I'm basically zeroing out all of the arrays in that and then over here basically getting a. I'm basically defining the variable called parameters and I'm getting the information about the arrays. So it's basically like the, like names of the arrays and stuff. And then I'm associating that with the zeroed out numbers that I did before. And so for each element, and so for the previous one, for each element E in the original list, it's creating this new array filled with zeros. And basically I'm using an association so that I can map the different perimeter names, which are the different elements of the perimeter, and their associated numeric arrays which are zero from before. And then now in the list new I have a list with altered data. So it's the parameters and then the zeros. And then I'm inputting that into this new model. So it's the same GPT2 model, but I'm using data from list new instead of the original data. And so I'm using NET replace part for that one. And so now what this line does is it basically it's like an altered version of the GPT2 model with zeroed out weights. And then I'm going back and I'm analyzing the GPT2 models responses when all of the weights are zeroed out. So I'm using the generate sample function again and I'm using 50 tokens so that it can create a pretty long response as well. And so over here I did the text. If I can take over the world, I would. And then I'm seeing its response. And as you can see all of the like, there are a lot of words that here that don't even make sense, or they're just like random words kind of scattered together and there's like random punctuation, stuff like that. And so I'm looking at the how many words are not in the English dictionary kind of as a metric to see how like the extent to which this is incoherent. And so as you can see, there are 13 words in the sample above that are not in the English dictionary. So that shows that the text is really irrelevant to the prompt and it's outputting just random characters, random words that don't relate to the prompt at all and that don't make sense. And so from this I'm able to see that the weights do have a significant impact on the GPT2 models output because without the weights then that affects its ability to actually process the input that I inputted, since the two don't relate at all. And I'm also able to see that it affects its like kind of its punctuations, grammar functions, and how it extracts its data to produce an output. And so I'm kind of looking at how the arrays are pretty similar to the human brain because the brain has a lot of individual structures that each play a role in how we think and how we output or how we say information and process it. And so similar to that, GPT2 also contains 196 different nodes that affect the model's behavior. And how when I'm zeroing out certain nodes or changing the weights or the impact that that node has on the model that affects its function overall. And so lastly, I'm looking at the temperature parameters, so, so I mentioned that as one of the items I wanted to study before. And so basically temperature parameters, those range from zero to two. And higher temperatures make the output more creative, whereas lower temperatures closer to zero make the model more predictable and generate more common words. And so basically when I was defining the generate simple function.

So when you say no problem, when you say temperature, you don't mean like actual temperature, it just refers to how creative or common the model becomes. Right?

Yeah. So with temperature, it's basically just one of the scales for the model's output. And so it ranges, it's like a higher temperature essentially means that it's more creative.

Okay. Yeah, thank you.

Yeah. And so basically I already included the parameters earlier for the generate sample function. And so all I have to do is I can just add the desired temperature at the end when I'm calling the function and I can see how that affects its output. So with the temperature of zero, I realized that the outputs are really repetitive. And so as you can see, like my output, my input was if I could take over the world, I would. And I used 100 tokens, so that would generate like a relatively long response. And I noticed that it was just generating like the same sentence over and over with a temperature of zero. So that basically shows that it uses the most predictable words based on its data set. And it isn't really creative or doesn't like add any other variations or different topics in this response. And I used the same thing for a temperature of 0.5. And I noticed that at first it's pretty, it's a lot more like creative than before, but towards the end, it's starting to, like, repeat the same phrase over and over with. I would create a world in each sentence. And the last thing, one of the things I looked at was a temperature of 0.8. And basically I realized that when I experiment with a temperature of 0.8, it's like the happy medium of the GPT2 model. And so you're able to see that it's a lot more relevant to the topic, it's a lot more grammatically correct, but it's not just simply repeating things over and over like it did before. And things that are temperatures that are above 0.8, they're not as repetitive as before, since they're a lot more creative, but it's also more irrelevant to the topic and it can start to grow more and more insane. So I looked at a temperature of 1, which is pretty much. It's a lot more higher than 0.5 and 0.8. And I'm able to see that it's not repeating any words at all. And it's a lot more creative, but it's also showing a lot of random information that don't relate to the topic at all. But the maximum is a 2. So I tried 1.2 as well. And I'm able to see that just at 1.2, it's already starting to reach its breaking point where it's starting to go crazy. It's generating outputs that don't make sense, kind of bringing in like random text from its training data. And so technically, you can experiment all the way up to infinity, but as you go higher and higher, the model will start to react that same way, showing the same trend and output text that doesn't really make any sense. So it's really similar to when I zeroed out all the nodes or I zeroed out all the weights within a certain node, then it would start to output information that was really incoherent. And so, yeah, that's basically my project.

Yeah, that was super cool. You know, there's all this, like, news about AI recently, and it was really interesting to see how you kind of break down an AI and like, just truly understand how its brain, I guess, works. So, yeah, I mean, it was really fascinating to learn about this. Do you have any future work planned for continuing this project onwards? Maybe like a different project.

So one of the things that I really wanted to experiment with was seeing how, like, the purpose of each layer in the model, even though there are a lot of layers, there's 196, but I wanted to see like, the function that each layer serves in the overall model and how like zeroing out one specific layer rather than just zeroing out all of them, like I did in this project, how that can affect its overall function. So like seeing if there's a specific layer that affects its grammar or its punctuation or like certain topics from its from the input that I give it. And so that's one of the things that I'm planning to look at in the future.

Yeah, that sounds really cool. Thank you so much for coming on and presenting your project. I definitely learned a lot and yeah, it's just really fascinating to see.

Of course. Thank you for having me.

Have a great day. Thank you all for listening to this episode and this season of the Wolfram Student Podcast. This episode concludes the end of Season two of our podcast. We hope you learned something new and took away insights from today's discussion. Please if you want to be featured in a future episode, we will be sending out the forms for season 2.5 in the spring and season 3, hopefully in fall of 2024. Once again, thank you all so much for listening to this episode and season of the Wolfram Student Podcast.