Elon Musk's xAI released Grok-1, a 314-billion-parameter model, as open source under Apache 2.0. The uncensored base model needs roughly 320 GB of VRAM to run and outperforms LLaMA 2 70B on benchmarks like GSM8K and HumanEval. It uses a mixture-of-experts architecture with 8 experts, only 2 active per token.
A week ago Elon Musk said that this week he's going to open source Groq and it actually happened today. They released the weights of GROQ1, the LLM that is powering charts on the X platform. But just like Mistral AI, they released the weights as a torrent file. Now the model itself is much bigger than expected. It is 314 billion parameter mixture of expert model and the model released is only the pre trained model. So this is the base model which was trained in October 2023. It's not the fine tuned version. That means it's going to be completely uncensored and it's released under Apache 2.0 so you can use this for commercial purposes. So here's a quick summary of the technical details. Thanks to Andreo. On X it's 314 billion parameter model. It's an MOE and for each token only two experts out of the eight experts are active. So that means for each token there are going to be around 86 billion parameter that are going to be active. It has 64 layers, 48 attention heads for queries. The internal embedding size is 6,000 and it's using the rotary embeddings, so it's using rope. The tokenizer that this model is using is very similar in size to GPT4 and it seems to have support for 8bit quantization as well as activation sharding. Now the max sequence length or the context window is about 8,000 tokens. Now this is a humongous model, so in order to download the model you will need around 300 gigabytes of storage space. Now GPU poor like us cannot run this model locally. So in order to run this in 4 bits you will likely need around 320 gigabytes of VRAM according to Imad who is the CEO of Stability. And in order to run this in 8 bit you will need DGX H100 which has 8 H1 hundreds and each one of them has 80 gigabytes of VRAM. Now in case if you have that kind of hardware and you want to run this locally, you can do that, they release a GitHub repo. In this case they provided a JAX example. So first you need to clone the repo, then run PIP install R requirements txt. This will download all the required packages. You need to make sure to put everything in this checkpoint directory. So you will need to download the weights of the model and then you can run the python run PY file so you have everything that you need to run the model. There's a really interesting code of conduct and it says be excellent to each other so make sure you follow grok's advice. Now if you're wondering whether it's worth downloading or not so on these four benchmarks which are the GSM8K, MMLU, Human Evolve and Math this GROQ1 model is better than LLAMA 270 build model almost on all the benchmarks and is also better than GPT 3.5. So it is a capable model but I don't know if it's worth having to use a DGX to just run inference on a local machine but since it's available under Apache 2.0 so it could be a great option for companies because of it its size because bigger models trained with enough data tends to be much better at reasoning so if you have a really good data set you can further fine tune this base model and probably get much better results and also it's completely uncensored anyways that's all the information that we have regarding the GROQ and its open weight release.
