LARRUZ compares inference speed on Groq and SambaNova by running Llama 3 3.8B and other prompts on both. Groq hit around 1,200 tokens per second versus SambaNova's 1,000 in these tests. Groq stays faster but SambaNova closes the gap while offering multilingual models and a free playground.
About 1,200 tokens per second.
About 1,000 tokens per second.
Llama 3, 3.8B.
2024-06-17.
LARRUZ.
Okay, over 1,000 tokens per second for a platform that is not Grok is pretty amazing when it comes to infant speed, Grok is pretty much the gold standard today. However, there is a new company called Sambanova which is also creating dedicated hardware that can get you over a thousand tokens per second. And just like Groq, they have multiple open source models on their platform. But unlike Grok, they are also creating their own foundation models along with their own dedicated chips like Groq. Okay, so we're going to do a speed comparison between Groq and the Samba Noa platform. Now, both of them provide Llama 3.8 billion model on their playground and that's the one that we're going to be using today. Apart from Llama 3, the Samba platform also has other models like Mistral and their own models which are dedicated to different languages. So for example, there is the Samba lingo, Arabic, Bulgarian, Hungarian, Russian. And this goal seems to be to create these multilingual models. Now, to do the speed comparison, we're going to be running the same prompt through both of these platforms. So first I'm going to start with Groq and we're going to use this example prompt, what is generative AI? And we get a pretty quick response. This is about 1200 tokens per second. Now we're going to run the same prompt and through Semblanova platform. And this is also pretty quick, but we are getting about thousand tokens per second, which is insane if you compare it to other offerings. But still, Groq is faster for this specific prompt. Next, I'm going to choose an example prompt from here. So let's use this draft. An email following up with the customer after an introductory sale call. And we consistently get about thousand tokens per second. What about Crocodile? So again, about 1100 tokens per second. And the responses are also pretty consistent across these both platforms because here it gives me an email add email, and then it says the email aims to thank the customer for their time and reiterate the main points of the call, provide additional information and resources that may be helpful. And even from Grok, it actually put this section at the end. The email aims to reiterate the value proposition and key benefits of your product services, outline the next steps to move forward with the proposal, and so on. Okay, now in order to test the real inference speed, you actually want to test it on much longer text or context. That's when the speed of generation is going to slow down a bit. So for this I'm going To be using this text, which is text from one of the letters written by Paul Graham which talks about how to do great work. So we are going to take about five pages worth of text because there is a limit on how much text both Grok and Samanoa lets you process on their their playground. And my prompt is going to be summarize this text. So I'm going to just copy the text and let's see how long it takes for Sambanoa to process it. Well, actually it was pretty fast. This is about thousand tokens per second, so this is pretty good. Let's run the same thing through Groq as well. And let's see, we still get about 1200 tokens per second. This is pretty amazing. Now in terms of the actual summary, so it starts off by saying we're looking at Grok. First the article discusses the commonalities among techniques for doing great work across various fields. Right. And it says that there are different recipes. So it came up with four different recipes. Choosing a field that aligns with your natural aptitude, deep interest and focus of focus for great work. Learning enough about the field to reach its frontiers. Noticing gaps in knowledge and exploring promising areas and boldly chasing outlier ideas even if others are not interested. Now Here are again four points from Samba Noah Lama. 3. Choose a field that aligns with your aptitude and interest. Learn enough about the field to reach its frontiers. So pretty much the same thing. Notice gaps and anomalies in the knowledge and explore promising ideas and pursue them. This is pretty interesting. The result seems pretty consistent across both GROK and Samanoa. GROK still is, I think, the gold standard when it comes to inference speed. But it's really great to see that there are other platforms which are generating text, I think at the same speed which are on par with grok. Having options is always great. This is probably the second fastest speed that I have seen so far in all my experiments. Now a couple of things. GROK actually provides a free API at least up till now, and you can interact with not only Llama 3, but there are other models which are listed here. So it's limited to four models. And GROK also recently added the ability to use vision models. So I'm actually looking forward to that. That's going to be pretty interesting. On the other hand, Summano gives you this free playground that you can play around with the models, but I believe they don't have any API which you can use for free. They do have some open source models of their own. That you can download on your local machine and start playing around with those, but their focus seems to be more on enterprises, so you'll have to sign up for a paid account in order to start using the API. Anyways, it's actually really great to have multiple options, and it's great to see that there are other companies which are pushing the frontiers of inference speed. I hope you found this video useful. Thanks for watching and as always, see you in the next one.
