Andrew Ng and Travis Addair from Predibase teach a course on serving large language models efficiently. You'll learn KV caching, continuous batching, quantization, and LoRA, then build an LLM inference server in PyTorch to see how different techniques affect performance.
I'm excited to introduce this new short course Efficiently Serving Large Language Models. Built in partnership with Predibase and taught by Travis Dare, this course presents a step by step deep technical dive into how the text generation process of transformer networks is implemented. This turns out to have a significant effect on the time to first token, that is after you've input a prompt, how long the user has to wait to get the first token of output as well as the overall throughput of an LLM. Travis, who's an expert on AI serving infrastructure, will show how LLM inference is actually made efficient and cover key technical details like KB caching, which caches computations from earlier steps to speed up the generation of later tokens. And you learn about precisely how the computational step steps are carried out. Whether you're serving your own fine tuned LLM or one that someone else has pre trained, this course will give you a much deeper understanding of what's actually happening under the hood. I'm delighted to introduce the instructor, Travis Zidair, Co Founder and CTO at Predibase. Travis leads an elite engineering team developing Predibase's platform for training and serving lms. Prior to that, Travis led the team that built Uber's Michelangelo's platform's deep learning capabilities and also led the development of popular open source machine learning frameworks Horovot and Morex.
Thanks Andrew. I'm really excited to be here. In this course you'll learn how large language models generate text one token at a time and how techniques like KV caching, continuous batching and quantization can be implemented to speed things up and optimize memory usage as you serve multiple users at once. And you'll learn how to implement your own LLM inference server in Pytorch by implementing these state of the art algorithms from scratch and measuring their performance.
This sounds great. And in addition to serving a single pre trained model, you also learn techniques like low rank adaptation or lora, which can be efficiently employed to serve hundreds of different custom fine tuned models on a single device without sacrificing throughput.
That's right Andrew. It turns out there's a lot more to it than downloading a model from hugging face and putting a web server in front. With the knowledge from this course in hand, you'll better understand the trade offs that you have to make as you think about the performance of your application and you'll be better positioned to evaluate what a potential vendor is offering you and whether their promises are realistic. This will help you make the best decisions for your project and for your company.
Learning the technical details of how OMs are served will help you become a better developer. I hope you enjoy the course.
