A YouTube walkthrough demonstrating Llama 3 models running on Groq Cloud’s playground and API, emphasizing extremely fast inference (hundreds to ~800 tokens/sec) on both 70B and 8B models. Shows live playground tests, a Google Colab/Python client example for API usage (including system messages, temperature/max_tokens, and streaming), and notes that the service is currently free with token rate limits.
Okay, so this is the actual speed of generation and we're getting more than 800 tokens per second, which is crazy. I haven't seen anything like this before. So since the release of Llama 3 earlier this morning, a lot of companies are integrating this into their platforms. The one I'm personally really excited about is Groq Cloud because they have the fastest inference speed that is currently possible on the market. Now they just integrated llama 3 both into their playground as well as the API. So now you have both the 70 bill and 8 bill version available. I'll show you how to get started with it, both in the playground as well as on the API if you're building your own applications on top of it. So we can select the Llama 3 models. So let's try stuff with the 70 billion model. Now I'm going to be using this prompt as a test prompt. We don't really care about how the responses are in this video. We only care about the speed of inference. So the prompt is I have Flask for 2 gallons and 144 gallons. How do I measure 6 gallons? And it probably has seen this prompt in its training data. Here's the speed of inference which was crazy fast. So it took about half a second. And the speed of Generation is around 300 tokens per second. And we are talking about the bigger sibling. So this is pretty great. Okay, now let's test the same prompt on the 8 billion model and let's see how the response is going to be. So this time it was about 800 tokens per second and it took a fraction of a second. So this is pretty great. Now let's see what happens if you actually ask it to generate longer text because as you as the model generates longer text, it's going to take more time. But let's see if it has any impact on the number of tokens per second. So here we are asking it to write a 500 word essay on the importance of open source AI models. First I'm going to use the 8 billion model. And here is an essay. Now the number of tokens per second is pretty much the same. This is pretty impressive. Next, let's look at the 70 billion model. And after this I'll show you how to use the API. So we're going to run this. Okay, this was real time speed. It's definitely not 5,000 words, but probably somewhere around a couple of thousand words. But the speed of generation is pretty consistent. So this is awesome. Now you can also include a system message if you want, you usually want to use the Playground to test the model as well as the prompts. And once you're happy with that and you want to integrate it as a part of your own applications, then you want to move on to Groq API so that you can start serving your users. Okay, so I put together this Google notebook to just show you how to use Groq in your own applications through the Groq API. So first we need the Python client. So we use pip, install Groq to install that. Next we need to provide our own API key. So for that go to the Playground, click on API keys, then create a new API key, and here you can create a new API key. I already have existing API keys, so I'm going to be using those. And since I'm using Google Colab, so I actually put my API key as a secret here and enabled access of that specific key to this notebook. So now we need to import the Groq client. We will create the Groq client using this Groq function and we need to provide our API key. So since I am reading it directly from the Secrets within Google Colab Notebooks, I'm using this user data function within the Google Colab API client. Okay, so this is how you set up the client, but let's see how you do inference. So it's pretty simple, straightforward. So we're going to be using the chat completion endpoint here we are going to create a new message. So right now we're just using a role of user. So the user is asking a question and the prompt is explain the importance of low latency LLMs. Explain it in the voice of John Snow. Now you can also add a system message later on. I'll show you how to do that. And then you need to provide the name of the model. Now, as of the recording of this video on the documentation page, the supported models lists only the Llama 2 family. However, Sandeep, who is the head of Grok Cloud, was kind enough to point out to me that it's available. So in this case I just used that model, the llama 370 build, and then following the exact same format, I provided the context length and this seems to be working. Probably by the time this video is released, they probably will have updated the documentations. So let me show you the actual speed of inference, which is pretty crazy. So I'm going to be running this. It will create this message, send it to the API, get a response, and then Python is going to print it here. But here's the actual speed of generation that we get. So you can see it's I think under a second. Pretty crazy that you can do this with an API. And we are running a 70 billion model. All right, next let's see how you add a system message. So here I'm adding a system message to the message flow. The role is system and we are saying you are helpful assistant. Answer as John Snow. So the rest of the prompt is exactly the same as what we did before. And we are selecting llama3 as a model. Now, you can also pass on some extra parameters. For example, you can set the temperature which will control the creativity or the selection of different tokens. And you need to also. Or you can also pass on max tokens that the model can generate. These are optional parameters. Now with that system role, here is the actual speed of generation. Again, this is pretty fast. Usually you want to add streaming so that the user is not just waiting for the response. But Grok has figured out how to do really insane the fast inference. Okay, but if you want to do streaming is actually also possible here. So the structure is the same as before. The only thing that you need to do additionally is to enable streaming by setting stream to true. In this case, we are creating a streaming client. Now, when you are streaming, then you are going to be getting a chunk of text one at a time. So we are basically getting that chunk of text, printing it to the output and then waiting for another chunk of text to arrive, showing that again, and so on and so forth. So here's the actual streaming speed. Let me run that again. Okay, so this is pretty fast and I think we all know that Gronk is known for this, but this is probably the fastest llama 3 infants that is currently available on the market. Okay, so a couple of other things. Both the playground as well as the API is currently available for free. So you can use this in your applications for absolutely free. They will probably introduce a paid version pretty soon, but since it's free, there are rate limits on the number of tokens it can generate. So make sure to look at this. Okay, so I'll be creating a lot more content both around Llama 3 well as Groq. So if you're interested, make sure you subscribe to the channel. I think they are also working on integrating support for Whisper on Groq. When that is implemented, I think that will open up the possibility of a whole new generation of applications. So I'm actually really looking forward to that. I hope you found this video useful. Thanks for watching and as always, see you in the next one.
