Gemma 2 2B: A Small Model Punching Above its Weight

This is pretty neat. Now we can understand the internal thought process of the LLM when it's generating responses. Yesterday Google announced Gemma2.2 billion model which is the smallest yet the most powerful model in its weight class and it already has some controversy around its performance. More on that later in the video. This model was released with this blog post titled Smaller, Safer, More transparent, advancing responsible AI with Gemma. This release contains three different models. One is the Gemma 2. 2 billion model which is a brand new version of their popular 2 billion parameter model. So you can run this on device and can be a good alternative to Apple Intelligence. The second one is Shield Gemma which is a suite of safety classifiers on top of Gemma 2. The third one is Gemma Scope which is probably one of the most interesting announcement and probably the most consequential one. So this is a new model which can be used as an explainability tool that offers unparalleled insights into the internal working of the model. So far the LLM has worked as black boxes, but now with tools like gemascope we will be able to understand the internal working and have more insights into how they are generating these responses. This is a model that is small enough that you can run it on device, especially on smartphones. But how good is the performance? Well, Google reported the Chatbot Arena ELO score and this is where things get very interesting. So this small 2 billion model is able to outperform much bigger models including the original Mixtral MOE. This is 8. 7 billion model as well as all the different variants of ChatGPT. We are talking about GPT 3.5 plus the original llama 270 billion model. So for a size of this model it's a very impressive performance. Now there has been some criticism of Chatbot arena lately especially I highly recommend to read this blog post GPT4OH mini change chatbot arena. And if you look at the model standing today, GPT4 or mini, which is supposed to be a much smaller model compared to other foundation models like Gemini or Claude. It's sitting right next to GPT4O, which is the state of the art foundation model from OpenAI. Now to understand why that is happening, we need to understand what exactly Chatbot arena measures. In fact, the ELO score basically measures human preferences, like how humans prefer the responses from one model over the other model. So it seems like the model creators are also tuning these models so that they generate responses which humans are going to like more, which is actually a good thing. Now Chatbot arena also gives you the performance based on some Other domains. If you look at the overall performance, GPT4OHM Mini is the second model or actually it's a top ranked model along with gpt4ohm. But for mathematics it's a little different story. Now if you look at coding, Gemma 2 billion model is behind the Mixrel Moe. In fact, if you look at the MMLU score, this gives you a much better picture. So here you can see that Gemma 2 is performing better than the previous version, but it's well behind the 7 billion parameter models. And that trends actually holds for all the other benchmarks as well. In fact, on human evolve, which is the code generation problem, it actually performs worse than the original GEMMA model. Now that doesn't mean that it's a bad model. It is created for a very specific purpose and that is human preferences. If you are running this on device, that is probably the application you are looking for. Okay, so some very quick technical details of the model. It is trained on 2 trillion tokens using knowledge distillation from a much larger GEMMA model. You need around 1 gigabyte of memory. If you're using 4 bit quantization, this has a context window of 8,000 tokens. And if you want to test out the model, it's available on Google AI Studio so you can select it here and start interacting with the model right here or use it through their API. It's also available on Hugging Face, so you can download the model and run it locally using their Python SDK. We will do a couple of tests using Gemma2.2 billion. So to test it out, I'm going to use this prompt which actually went viral over the last few weeks. What's larger, 9.9 or 9.11? So let's see what the response of this model is. So it says 9.9 is larger than 9.11, which is correct. We're going to ask it to explain its reasoning. So you says you're right to ask for the clarification here. Numbers are ordered. Numbers are arranged on a scale. Larger means greater. Right? So it knows this then 9.9 is greater than 9.11. That's because 9.9 is further to the right on the number line than 9.11. Now when I asked the same thing from Llama 370 billion model, what, what's larger 9.1, 9.9 or 9.11 it came up with this answer. Now it has a lot to do with the tokenizer that is being used and, and also the context that you need to provide. Okay, just to see if that's not directly coming from the training data. So I said which is larger, 7.77 or 7.33 and it still gets it right, which is pretty awesome. Now you can also use it for something like sentiment analysis. Now it's a good model for conversation, probably for something like programming. It's not a great model as shown by the benchmarks here. Let me know if you want me to do a more comprehensive testing. If you want to run this locally using transformers, you can do this well. And here's a simple code snippet that you can use. So you will use the transformer package. You will just download the tokenizer, then load the model which is going to be google gemma2 and then just provide your prompt and use the tokenizer to convert this prompt to the corresponding tokens. And then use the generate function on top of the model with the max number of tokens that you want to generate. This will generate the final response. So again, if you want me to do a complete video of testing, let me know. I will put something together. Before wrapping this up, I also wanted to talk about these other two models which are I think equally important, especially the Gemma scope 1. First let's look at Shield Gemma which is a safety classifier. Now you can use this to process both the input from the user which is the prompt and as well as the responses generated by the LLM or the outputs. So you can use it to filter hate speech, harassment, sexually explicit content as well as dangerous content. This is an integral part of any production ready system that is user facing and it does achieve state of the art performance for iterate class. Now the Metal Llama also has a number of different models for content filtration and safety. The second or the third release is gemascope, which is the one I think that's going to have the most impact because right now these models are just black boxes. We don't really know what exactly goes on, but the gemascope gives us an insight into the internal working. So this is a number of sparse encoders that Google is releasing. There are over 400 freely available sparse encoder which are going to be covering all different layers of Gemma 2, 2 billion as well as 9 billion models. So here's a quick demo of how this works. I'll put a link to this website. Highly recommend to test it out. So here's the user input prompt and you can select a specific token from the input and then see that what the model thinks. So these are different features that the model has learned things about these are different categories. So whenever you select a token, for example something like let's say execution, so it's titles and references related to high level leadership positions or executive roles terms related to generic genetic testing and analysis. Right. So these are the different parts of the model that are activated whenever you select a specific token. They also provided a complete Google Colab notebook that you can use to run these sparse encoder and look at the activations of different features based on the user input prompt. So I'm going to put a link to this. I'll highly recommend to test it out. Probably create another more detailed video on this because it's a very interesting topic. So this was a quick video on Gemma 2. It's actually very exciting because there is a push for smaller models. So Apple is also releasing their Apple Intelligence which are a number of different smaller models. There is also supposed to be Gemini Nano, which is a class of model that is supposed to be powering some of the intelligence on Google devices. So Gemma 2 is definitely really a great model that can be run on device. So we'll see how this pans out. I'm also excited about Gemma Scope, which I think is going to bring a lot more value for our understanding of how these models work internally. I hope you found this video useful. Thanks for watching and as always see you in the next one.