RedPajama - Llama is getting Open Source!

In this brief video we will discuss an exciting project known as Dirt Pajama, initiated by a group called Together Compute. The aim of the project is to create an entirely open source version of the LLAMA models. They have started by releasing an impressive data set of 1.2 trillion tokens. Their goal is to develop a series of open LLAMA models by training foundation models on open over a trillion tokens. The data set they have released is based on the original LLAMA models consisting of 1.2 trillion tokens. For comparison, Llama's 7 billion and 13 billion parameter models were trained on 1 billion tokens, while the larger 65 billion parameter model was trained on 1.4 trillion tokens. This release is significant considering the extensive pre processing and cleaning involved. In theory, this should allow the development of a model as effective as llama. Many large language models such as Llama, Alpaca, Oconia, Koala are not entirely open source, although some like Patia, openchatkit, Open Assistant and Dall Er Redpajama aims to create a fully open llama model. As they put it, this is the Linux moment for AI. A number of international groups are collaborating on this project, including individuals from Stanford Eth, Switzerland and Mida in Canada. The project involves three main components. The pre training data so high quality and broad coverage. The base model which are currently being trained. An instruction tuning data set which is to be released later. Now let's talk about the data set itself. So the data set comprises of five dumps of common crawl clean using various filters. It also includes the C4 data set, GitHub archive papers, the Books Corpus, Wikipedia and Stack Exchange. You can potentially download it and use it on your own machine, but you will need a very huge hard drive. So the full Data set is around 5 terabytes unzipped on the disk and 3 terabyte to download compressed. The data set is available on Hugging Face. So if you have the resources you can train your own demo model. So here is a preview. They also provide a smaller sample data set containing a billion token for easier exploration. So you can find it here on this link and experiment with it if you want. Apart from the dataset, they are also releasing a dashboard where you can explore the data set and play around with the embeddings. I'm really excited about the embeddings part because you can then use this with something like LangChain. So in conclusion, the Red Pajama project is well on its way to producing a fully open source lamap model which could potentially replace other models like Vicunia and Koala in near future. If you have any questions, feel free to leave them in the comments. And don't forget to like and subscribe for more updates. Thanks for watching. See you in the next one.