Yandex's PV-Tuning: Run Llama 70B at home! Local AI just got more accessible! 🚀🦙

LLMs are incredible, right? Yeah, incredibly large. Most consumer level GPUs can't even run these things well. What if I told you that large language models are about to go on an incredible diet? Introducing the PV tuning method. This is open source tech developed by Yandex Research in collaboration with IST Austria and Cost and they are revolutionizing the way that we compress large language models. PV tuning was applied to Llama 3.1 and the results are mind blowing. Llama 70B was compressed 6.4 times from a whopping 141 gigabytes down to just 22 and it still retains about 93% of its original quality. That 70B model can now run on a consumer grade 3090 GPU. The already small Llama 8B has been shrunk so much that now it can run on an Android smartphone with just two and a half gigabytes of ram. Like I said, it's all open source, so if you're interested, links to the hugging face models down below.