Andrew Ng and Hugging Face teach how to shrink massive language models. A 7B parameter model takes 28 GB uncompressed, but quantization can cut that dramatically by converting floats to 8-bit integers. Hands-on labs use Transformers, Quanto, and PyTorch.
I'm excited to introduce quantization fundamentals with Hugging Face. Developed in collaboration with hugging face, generative AI models can be huge. For example, even a modest 7 billion parameter large language model uncompressed might take 28 gigs just to store and this.
Makes them hard to download, store and.
Run on consumer grade hardware. Quantization is a critical part of the LLM and more broadly the AI software stack for compressing large models and making them more accessible to everyone. In this course you use the Hugging Face Transformers library, QuantumLibrary and a bit.
Of Pytorch to load and run models.
Using data sets such as Integer or Int8 as well as floating point representations such as Float16 and bfloat16 which stands for Brain Float16 since it was created by my old team, the Google Brain team.
You also learn about the technical theory.
Behind quantization of say how to compress.
A 32 bit floating point number into.
Maybe 16 bits and you learn to.
Apply linear quantization yourself to compress a model. By the end of this course you have a good understanding of the methods.
And challenges when quantizing machine learning models. I'm delighted to introduce the instructors, Eunice.
Dakodar as well as Mark sun, both of whom are machine learning engineers at Hugging Face, deeply involved in developing and making open source models widely accessible.
Thanks Andrew. In this course you will first learn about basic concepts around integer and floating point representations and how to load AI models using different data types. Using Pytorch and the Hugging Face Transformers library, you will also dive into a technique called linear quantization. You will use it to map floating points which can range from very high to very low values, into eight bit signed integers which can only represent numbers from -128 to 127 using linear scaling and shifting factors.
Next, you'll directly apply linear quantization into a small open source generative model using the Quanto library from Hugging Face. And we will wrap up the course by giving a detailed overview of the current status of state of the art quantization applied to large language models.
We hope this course will give you good insight about basic building blocks around quantization and you will be able to directly apply the tools we are going to use into your own project and use cases.
Finally, I want to leave you with.
One last thought which is that model.
Compression is no small feat.
I hope you enjoy the course.
