World's First Language Processing Unit 🚀 🚀 🚀

Okay, this is going to blow your mind. On the right hand side we have GPT4. On the left hand side we have Mixtral 87B. Now look at the speed of generation for the same prompt and this is real time. Okay, this was able to generate almost 500 tokens per second. GPT4 is still processing a request which is insane. Now let's try the same thing with GPT 3.5. Here is Grok and this time it generated text with almost 500 tokens per second. Again, this whole generation took around 1.68 seconds. ChatGPT is still generating text. There is this new company called Groq which has created LLM specific hardware and the speed of generation for infants is absolutely insane for open source large language model, it's able to give us the best possible inference speed of all the platforms that are offering inference services. And all this is possible due to their new hardware platform which they are calling LPU or Language Processing Unit. This is dedicated hardware developed for LLMs. Compared to the best available GPUs on the market, this delivers 18 times faster inference for LLMs, which is absolutely insane. The best part is that you can try this yourself and they also have API access at an extremely reasonable price. Currently GROK is hosting a demo that gives you access to llama270 build model and the MOE or mixture of Expert model from Mistral AI before looking at the speed of inference of groq. Let's see who is behind this. Jonathan Ross is the CEO and co founder of Grok and is also the creator of First Language Processing unit which they are calling lpu. Previously he was at Google and is the creator of TPU or Tensor Processing Unit. That is the Google's dedicated hardware for deep learning. So let's look at a couple of demos before looking at how they're able to achieve such amazing inference speed. And later in the video we're going to also talk about the API pricing if you are interested in using their services. So as I said in the Beginning, there are two models currently available. One is Llama 2 and the other one is the mixture of Expert from Mistral AI. My goal for this video is just to look at the speed of inference rather than the accuracy of the responses. So that's why we're going to be using just a single prompt to look at the number of tokens that are generated per second. So let's first try Llama 2. This is the slowest of the two models that are available on Grok. Here's the example Prompt that I want to use write a new chapter of the Game of Thrones where Jon Snow is giving his opinion on iPhone 14. And initially we will use the llama 270 build model. This is real time speed. So it was generating around 280 tokens per second, which is absolutely insane. I haven't seen this kind of speed on any GPU. So the generation took around 2.08 seconds. Now if you click on this scroll to top, this will take you to the top of generation. And here you have a few options that you can play around with. So for example, you can ask the model to regenerate that in bullets. This is kind of a summary. You can also ask the model to expand on it. And it's constantly getting that 280 tokens per second for llama 2. But for the mixed role model, it's even crazier than this. Also you can ask it to, let's say just give you everything in three sentences, right? So these are different options that you can play around with. Now this platform also gives you the ability to set the system prompt. So here you can provide the system prompt if you want. And you will also be able to change some of the settings of the model generation. So for example, you can set the maximum number of tokens, the maximum input tokens, set the temperature as well. Currently it's just set to 0.2 top P and top K as well. Next we're going to use the Mistral mixture of expert model. This new hardware architecture seems to be well optimized for MOEs because you get much better speed. So if we run the same prompt through the system using the Mixtral model, now we are getting 480 tokens per second, which is a crazy amount of speed. This whole thing took only 1.62 seconds to generate. This is the best speed I have seen for LLM inference so far. Now before looking at how they are able to do it, let's look at the API access and pricing. So currently you can apply for API access and they are giving API access to approved members. Right now they have two models. One is the llama2.7 billion and the other one is the llama2.70 billion model, both with a context window of 4000 tokens. The mixtral MOE is going to be available soon and you can request for that. Now the good part is if you get Approved, you get 10 days free access where you can use up to 1 million free tokens. And the API is fully compatible with the OpenAI API. Currently they are opening access to alpha users so you can apply for access. There is a request access form. You just need to provide your basic information, email name, organization, title and which industry you want to use this for. I have submitted my request and will update everyone when I get access. General availability of this API is going to be in Q2 2024. Now, in terms of the pricing, according to them, Groq guarantees to beat any published price per million tokens by published providers of the equivalent listed models. And this seems to hold true. Specifically if you look at the Mixtral MOE model on the Mistral AI website, which I believe is this Mistral small, the input tokens are 0.6 Euros per million token, the output tokens are 1.8 Euros per million token and here you are going to get that for $0.27 per million token, both for input as well as output tokens, which is pretty amazing. And the speed of generation or inference that you get here is blazing fast, which opens up so many possibilities. So when Groq API access is available, I'm definitely switching to this. Okay, now let's talk about how they are able to achieve such amazing speed. If you look at the list of competitors that GROQ has, everybody is using state of the art GPUs for inference, but in contrast GROQ is using their special purpose language processing units. So everything comes down to the architecture GPUs or graphic processing units that everybody is using for LLM inference were not actually designed for this purpose. GPUs were initially designed for running graphics intensive games, but later on they were repurposed for training deep neural network models because of their ability to parallelize computations and people also started using them for training LLMs as well as for inference. However, the nature of inference of LLM is way different. That's why lpu, which stands for language Processing Unit, is able to beat GPUs because it's a new type of end to end processing unit that provides the fastest inference for computationally intensive applications with a sequential component to them. And this sequential component is important for LLMs. So in their FAQs, Grok team talks about two main bottlenecks for LLMs. One is compute density and the second one is memory bandwidth. Now since LPUs are dedicated hardware for LLM inference only, so they have greater compute density compared to the available GPUs and CPUs and this greater compute density results in the reduction of time per word calculated. So that allows a sequence of text to be generated at a much faster speed compared to the traditional GPUs now one limitation the architecture has that it's great for inference however it's not good for training at all so for training LLMs you will still need to use GPUs however most of the time is spent on inference if you are looking at consumer facing applications and Groq is probably the best option if you are looking for inference speed at the moment anyways have a play with the platform. It's absolutely amazing. It's blazing fast and because of this speed of interference it opens up so many possibilities for different applications that were not possible before. For example you can have near real time conversation with an LLM now if you are able to convert the text generated to speech and you can use a faster whisper model to convert your speech to text. So this is going to open up so many different applications and I'll probably make subsequent videos when I get access to the API. I hope you found this video useful. Thanks for watching and as always see you in the next one.