New course series with Flower Labs: Federated Learning

I'm delighted to announce two new short courses, Intro to Federated Learning and Federated fine tuning of LLMs with private data. Built in partnership with Flower Labs and taught by Daniel Boytel and Nick Lane, federated learning allows models to be trained across multiple devices like phones or multiple organizations like multiple hospitals, without the need for anyone to share their data. Take the example of many hospitals each having private patient data. If a group of hospitals wants to work together to train a model from all this data but can't share the data directly because of patient privacy, federated learning is a great option. Instead of sending all the data to central server or cloud service, federated learning allows each organization to train a local model using their data. Once this local training is complete, only the model updates and not all the raw data is sent back to the central server. This ensures that personal data remains private, enhancing privacy and security while also letting us deploy models that benefit from having many inputs. This technique is also used for training nix web prediction models, at least on some mobile keyboards. How can we train a model across lots of users mobile devices to learn to predict what someone is likely to type next, but without needing to aggregate everyone's keystrokes, which is incredibly sensitive data, onto a central server? Google had pioneered a lot of the work on federated learning, motivated specifically by this use case training on hundreds of millions of Android phones and other large companies like Apple and Meta have also announced using federated learning for other applications. Federated learning allows you to build more powerful machine learning models while strengthening how you preserve confidentiality of the data. In these two courses, you learn about federated learning using Flower, which is a popular open source framework with a large community. I'm delighted to introduce the instructors Daniel Boytel and Nick Le. Daniel is co founder and CEO at Flower Labs and is also one of the creators of Flower, and Nick is co founder and Chief Scientific Officer at Flower Labs and also a professor at the University of Cambridge. Daniel, Nick and their collaborators have grown Flower into a vibrant community of thousands of federated AI researchers and developers.

Thanks Andrew, I'm excited to be here and to share more about Flower and federated learning with you. In the first course, Intro to Federated Learning, you'll explore federated learning and how it enables you to train machine learning models on distributed data. You will use the Flower framework to build a federated learning system. Specifically, you will learn how the federated training process works, how to tune and customize the federated training process, how to think about data privacy in federated learning, and how to reason about bandwidth usage in federated learning.

In the second course, Federated Learning Fine Tuning of LLMs with Private Data, you will apply federated learning to training LLMs. LLMs present a unique challenge in that they can in some circumstances memorize training data. LLMs also contain billions of parameters, making distributed learning more challenging. LLMs also require large amounts of trained data. You will learn techniques that build on federated learning which improve data privacy while still benefiting from the powerful capabilities of LLMs. First will describe how to apply federated learning to LLM training. Next, given the large model size of model LLMs, you'll also learn how parameter efficient fine tuning is important for making the training process much more efficient by decreasing the bandwidth needed to make federated learning practical for LLM training. Finally, we'll add on the technique of differential privacy. This protects individual training points, making it difficult to identify them as coming from a specific source. Each one of these techniques provide a key advantage. Together they create a powerful approach to federate the fine tuning of LLMs.

So through these two courses, I hope your ability to train great models while enhancing privacy will blossom. I hope you enjoy both courses.