Is Groq's Reign Over? Cerebras Sets a New Speed Record!

Lets see how fast this is compared to Grok. 1800 tokens per second, which is much, much faster than Grok, which is a surprising thing to say. So far, Grok was the gold standard when it comes to inference speed, but it changed a couple of days ago when Cerebras introduced their Cerebras inference. They are also known for their custom hardware for LLM inference, but unlike Groq, they have been training their own models or models for their customers. Their inference endpoint can provide up to 450 tokens per second for the 70 billion version of Llama 3.1, which is 20 times faster than H100 GPUs on hyperscale clouds and 1/5 of the cost. To showcase an example of their fast inference speed for LLM, they released this.

Hi there, how are you doing today?

I'm good, how are you?

I'm functioning properly, thanks for asking. I don't really have feelings like humans do.

Can you tell me what is the fastest API endpoint available for LLM inference today?

You're asking about Live Kit's APIs? Our local noise machine LNM inference Endpoint is a premium feature. For the fastest available endpoint, I'd recommend our real time. You're welcome. Have a good day.

They connected their inference API endpoint with LiveKit, which you can use to build voice assistants. You can also use Verbie to do the same. That's my own open source project. We're going to have a direct comparison with grok, but before that let's look at some technical details. They claim the new wafer scale technology will give you the world's fast, fastest inference. That means you get twice the number of tokens compared to GROQ, or 20 times the number of tokens compared to an H100. And because of that high throughput they are able to achieve lower cost. But I think the most important part is that they are able to do the inference in full 16 bit precision, which is not the case for most of the other inference API providers. And there is research which we're going to look at later in the video which shows that the precision at which you're doing inference is extremely important. They kind of broke the plot from any scale. So here on the X axis we have output speed, output tokens per second, and on the Y axis we have million tokens per USD. You can see that this gives you the best possible inference speed on the market today. Well, as the cost is extremely competitive. In fact it's only $0.6 per million token for NEMA 3.170 billion. Cerebrus also has some very impressive people behind them. Next, first we will look at the inference speed between CROC and Cerberus inference. And later in the video I'll show you the accuracy of inference, which also is pretty surprising. So in both cases, we're going to start with the 8 billion model. Here I'm going to select the Lama 3.18 billion instant, and the prompt that I'm going to be using is going to be to write the name of all the US governors of all states from 19, 19, 20 until 2024. I want to see how many max number of tokens both of these models can actually generate. So if we try this, we can see we already ran into context limit issues. The number of tokens that we are getting are 750 tokens per second for the 8 billion model, something which you want to pay close attention is to the number of input tokens, which is 65 in this case. And the output is limited to 2048 tokens. And that's why we don't really get the whole list, because the actual list is well beyond 4,000 tokens. Now, we're going to run the same test on the Cerebras inference. So we'll select 3.18 billion. Here I'm going to paste the same prompt. We're going to run the test again. Now, again, I don't think it gives us a complete list because it already reached the context limit or the maximum number of tokens that they allow, but it gives us an impressive speed of 1800 tokens per second, which is more than twice what Groq was able to do. In this case, it's using 45 input tokens. So I think the prompt template that Groq and Cerebrus are using are very different because there is a difference of about 20 tokens. It also gives you almost twice the number of output tokens compared to Groq, especially on the chat interface. Now, when it comes to the API, it's a different story over there. Cerebras limits it to about 8,000 tokens on the free account, which I believe Groq does a very similar thing. Next, we're going to try the 70 billion version. So we're going to use the same prompt again. Again, we reach the context limit and this is giving us 250 tokens per second. Again, it's using 65 input tokens compared to 45 that we saw in Cerebras. Now, if we run the same test to Cerebras, I think we're going to see very similar results. Now in this case, for some reason it says what a monumental task you asked of me. And instead of giving us the whole list, it just gives us a few resources where you can get the information from. But in terms of the number of tokens per second that it can generate is almost twice that of Groq. Now, with a little bit of prompt engineering, just by telling it, yes, you can give me a complete list, we get a list. Now this is not complete because it also runs into the max context that it can generate. But at least we get a list very similar to what we're getting from grok. The question would be why would the same model which is served on two different platform will give us different results for the same prompts? So it can depend on some of the hyper parameters that they have set. Also it can depend on the quantization level that is being used, which is a topic the Cerebras team has tried to address. And it's actually a very important topic if you are putting any LLM in production. Okay, so they have this blog post titled Llama 3.1 Model quality evaluations and they are comparing cerebras hosted LLAMA 3.1 with GROQ together and Fireworks AI. According to this blog post, not all LLAMA 3.1 models are created equally. And there are a couple of very interesting papers that they cite in here. And these paper shows that the quantization level that you're using can have severe impacts on the performance that you expect from these LLMs. So they ran different benchmarks with the APIs from these different providers. These plot shows Llama 3.170 billion instruct and 8 billion instruct served through different API providers, including Fireworks together AI, GRO and Cerebress together AI and Fireworks do use a quantization scheme. GROQ is probably also using some sort of quantization, but we don't really know the details. But for the same models served through different quantization levels and different providers, you actually see some stark differences. For the 8 billion model, you see that the 8 billion llama 3.1 served through together AI consistently performs worse compared to other API providers. And for some of these benchmarks there's a kind of a trend. So for example, if you look at the MMLU or the math benchmarks, then Together AI and Groq is doing worse compared to the Cerebras inference for code evaluation or coding related benchmarks. This is I think more clear. There's a consistent trend that the other models are performing worse on the same benchmarks. Although they are serving the same models, they also did evaluation for multi turn conversations. If you look at the single turn conversation together AI and Grok are doing pretty bad. For multi turn conversation they are doing better. We are talking about the 70 billion model but for the 8 billion version it's actually the opposite. So for singleton this model is doing pretty good even if we serve with two Groq together or Fireworks. But for the multi turn conversations there is a drastic reduction in performance when it comes to Grok together AI and Cerebras. These results shows that the same models serve through different quantization level and different inference hyperparameters can give you very different responses and hence different results on evaluation metrics. So you definitely want to consider these things if you are putting these models in production. Okay, at the end of the video let's talk about the API. So Cerebras also serve these models through the API you are limited to a context window of 8,000 tokens and that is because they offer 8,000 tokens context window for free tier the API standard is the same as a standard so this can be a drop in replacement. Now personally I haven't got access to the API yet, so that's why I'll not be able to show you any code examples. But if anybody can hook me up with an API that will be highly appreciated. You will have to join the wait list for this. I have already done this so if there is anybody from Cerebras watching this video please reach out. It's great to see there is competition when it comes to inference speeds. Higher inference speeds definitely opens up a lot more possibilities, especially for real near time interactions. Grok was the leader in this space and hopefully they will come up with a new update. Anyways, I hope you found this video useful. Thanks for watching and as always see you in the next one.