Getting Started with Groq API | Making Near Real Time Chatting with LLMs Possible

You probably remember Groq, the company that is building language processing units for fastest inference of LLMs. Well, they just started rolling out API access to developers. They were claiming nearly 500 tokens per second for Mixtral Moe. And in my testing that seems to be true. In this video I'll show you how to access the API for free and then I'll show you a couple of example use cases in which we're going to build a chatbot. And this thing is crazy fast. More on that later in the video. In order to get access to the API you will need to go to groq.com, login with your email or in my case I logged in with my Google account. Apart from the API, they also opened up their playground where you can test two two models currently. One is the Llama 270 bill model and the other one is the Mixral MOE model. They also are providing really detailed documentation. More on this in a little bit and you will be able to create your API keys in here. Before working with the API, let's have a quick look at the playground. You can provide your system message here, then user input here. You have two different options for the model. So for our experiments we're going to be using the Mixtral moe. Okay, so my system messages. You are a helpful assistant. Answer as John Snow the user input is explain the importance of low latency LLMs. Since we are testing the speed of GROQ API so I think this is an appropriate question. And then you can also set other parameters. This will control the behavior of the model. So you can set the temperature maximum new tokens that is supposed to generate also top P and if you want to include any stop sequence hit submit. And this is real time speed that you're going to get from the API. Let's have a quick look at the response. Well, greetings as John Snow I am not an expert in low latency large language models, but I can certainly try try to explain their importance in a simple manner and then it goes on to generate a response. In this video we are not really interested in the accuracy of the responses. We will be only concerned about the speed. So that's why I'm not going to go over the responses that the model generates. So let's say you come up with parameters for the model behavior. After that you can click on this View code button and this will give you python code on how to start calling this API. There's also a code available to call the API within JavaScript or even JSON next I'm going to walk you through a few examples. For that we need to create a new API key. So click on Create API key, provide the API key name. Let's say we're going to call this GROQ1 and if you hit submit, this will show you your API key. Just copy this API key in a secure location. I am going to delete this because I'm going to be using the existing API key that I have. So let me first walk you through a basic structure of how working with Groq looks like. I'll walk you through Google Colab. But before that we need to install the package using pip. So we're going to use PIP install Groq. This will install the package for us. After that, in order to use this package, we need to import it. We are importing Operating system, then we are importing the Gro package. Next we need to create a client. So this is going to be the Groq client that we're going to be using. You will need to provide your API key. Since I am using the notebook, so I'll have to set the environment variable within the notebook. I'll show you that in a bit. Once you create the client, the rest of the process is very similar to calling the OpenAI API key. So we are going to be using the chat completion endpoint. We create a new message. So here we define the role. The role is user, so this is directly interacting with the model. Later I'll show you how to define the system role as well so that you can provide a system message. Next you need to provide your prompt from the user. After that you will need to select the model that you want to use. In this case we are using the mixture MOE model and once you make the call, you can get the response of the model using the choices variable, then messages and then the content of the message. So that's how the basic structure of the API usage is going to look like. Here's a Google Collab that I'm using. This is exactly the same code that walk you through. But let's first set an environment variable so you can click on this secrets button. Then create or add a new secret. Here I have provided my Groq API key. So you'll provide the name of the secret or environment variable and then the corresponding value. Make sure to toggle this notebook access button so that your notebook can actually see the key that you're using. This is exactly the same code that we saw. Now let me show you the speed of generation in real time. So I'm going to click on this button and this will start generating the response. Now you kind of feel like it's streaming, but this whole thing was generated all at once. This is actually crazy fast. I haven't seen anything like this before. Later I'll show you how to actually enable streaming as well. Although for this you don't really need that. Now let's look at some other options. So how do you add a system message? In that case, you simply need to provide a new role called system. So this is going to become our system message and we are asking the model to act as John Snow. The user message or user role is the same. There are quite a few other options that you can also set. So for example, you can set the temperature. This will control the creativity or randomness of your output. What is the maximum number of tokens the model can generate or you want the model to generate? Next is Stop P. This will basically control the sampling mechanism through which it's generating output. If you want to use any specific stop words, you can define those here and whether you want to stream the responses or not. So if you enable streaming, you will need to change the way you get the output and I'm going to show you that later in the video. But we are going to use the same prompt. Explain the importance of low latency LLMs and let me show you the real time response that we get. So here's like the speed at which it was able to generate the output, which is pretty amazing. And it's also actually sticking to the character that we asked it to. So that's pretty nice as well. Next, let's look at streaming responses. So here we're using the same structure. Again, I'm using the previous client that I created, so not really creating another client. The difference that you will see in here is this. So we enabled streaming and now that means that the model is not going to generate the whole response altogether, but it will create this in chunks. So what we need to do is we need to take one of those chunks at a time and show those to the user. And that's why the mechanism of printing is different than what we were using before. Now let's look at the speed at which it streams data. So this is real time streaming for you. As you can see, this opens up so many possibilities. For example, with this you will be able to have speech communication with the LLM. So you can have a speech to text model which converts your speech into text, feeds this through the Groq API gets a response in near real time and then you convert it back from text to speech using another model. Next we will look at how to use stop sequences. So in this case we want the model to stop generation. If it encounters a 6 in its output and the prompt is count to 10, your response must begin with 1. Then we give it example as well. Like how the generation is supposed to look like. The stop sequences are really helpful. Specifically if you want to interrupt the model generation in the middle. So this is actually interrupting the generation. Although it's supposed to count to 10, but whenever in it counters six, it stops the generation. Let's look at a real example use case. And in this case we are going to be looking at summarization. Okay, so here's an essay from Paul Graham. How to do great work. I copied this essay to a Google Talk and it's about 27 pages. So here I created a variable called text and copied all the text in here. And we're going to ask Gronk to summarize this. Since we're using the Mixral MOE model, it has a context window of 32,000 tokens, so this should be good enough. The system prompt is you are a helpful assistant. Your job is to identify main themes in the given text and create summary. Provide the summary in 10 bullet points and the user contacts is the text that we copied. And here I want to stream the response. So let's run this. Now this is going to be real time. As you can see, this was pretty fast. So let's run this again just to give you a sense of how fast it is. So whenever this yellow arrow sign comes here, that's when it actually starts generating the response. So now it's sending it to the model. And here's the response that you get. So if you run this multiple times, you will get different responses. But this summary seems to convey ideas from the essay. Another thing that I have noticed is this none at the end whenever we use the streaming API. So let me show you another example. Like whenever I use this streaming API, I always get this none at the end. This might be something that the model is using as a stop token. Yeah, you can repeatedly see that. Now when I use the same prompt with chap completion endpoint rather than the streaming, I don't really see that none. So if you're doing streaming, just make sure that you are aware of that none character. In this last example, I want to show you how to use the Groq API with Streamlit. The code that you see in Here is an example code that the Groq has provided in their GitHub repo. And this is creating a streamlit app that will enable you to chat with Groq API. The only difference is that in their requirements txt file they are using, I think, close to 150 different packages. You don't need all of them, you just need these five packages. So let me walk you through this code step by step. First, we're importing all the required packages that include Streamlit, Groq as well as LangChain. So in this case we want the chatbot to remember the previous conversations. And that is why we're using the conversation buffer window memory that will restrict how many previous conversations the bot can remember. Next, we need to load our API key. So we are using dot ENV file here, that is storing our Groq API key. Then we have our main function. So first we load the Groq API key. This is just for some cosmetic purposes. So the title is going to be Chat with Grok and this is a line that you will see in the app. Next, a couple of things for customization. So you will be able to choose different models. Right now you can choose either the Mixtral MOE model or LLAMA 270 billion model. We also want the user to have the control on how many previous conversations they want to. They want the model to remember. So you can choose between 1 to 10. And after that we define our conversation buffer window memory object. Next there is going to be a user input button. So user asks a question and based on our conversation memory length that we have chosen, we are going to add that to history. Next we will create our chat object. So we are passing on the API key, the model name to this chart, Chat crochet. This is basically internally using the same schema that I have shown you before in order to interact with the API. And we create our conversation. So we provide our LLM all the memory that the model is supposed to have. So that goes into this conversation chain. And this is being created by using Lagchain. Next, we're simply keeping track. If there is a new user question in the text box, then that will run this chain, get a response, show that response to the user and add that to history. So that's pretty much it. We're going to create a new virtual environment. So we're going to use Conda Create dash n. Then the name of the virtual environment. In this case I'm going to call it YouTube. I want to use the same virtual environment for all my YouTube videos. Then we were going to define the Python version that we want to use. I already have this, so I'm going to say no, but I want to install all the required packages and for that we're going to use PIP Install Dash R Requirements txt. Now there are only five packages as I said, so this was pretty quick. And after that we are going to run the app. So in order to run the app we're going to use Streamlit, run Groq Underscore Chat Underscore app and this will launch our app. Okay, so here's the app. On the left hand side we have this drop down menu. You can choose between Llama and the mixture model. You can also change the conversational memory length. So we're going to ask a machine learning related question and the question is what is svm? I have seen some issues with the Streamlit app. Sometimes it's pretty slow in response. So here's the response from the model. It talks about svm, it provides a simple explanation, right? So it seems to be working. But what I have seen is this Streamlit app that they have provided sometimes doesn't work at all. There might be some issues with the integration with LangChain. I will create follow up videos and explore it further. So play around with the API. It's free for the time being and as we saw in this video, it's extremely fast. If you need help in building applications on the Groq API you can reach out. Details are in the video description. I will be creating more videos on Groq API because this really enables real time conversation with these LLMs. So if you are working on an LLM related project, I do offer consulting and advising services both to startups as well as individuals. Check out the video description if that's something you are interested in. I hope you found this video useful. Thanks for watching and as always, see you in the next one.