A short YouTube interview (published/recorded Aug 2, 2023) in which Andrew speaks with Cary Felt, founding product manager at Weights & Biases, about a short course titled “Evaluating and Debugging Generative AI.” Cary outlines course goals: monitoring and debugging experiments, tracing and evaluating generative models, visually comparing outputs, and adopting reproducible, collaborative workflows for model development.
Questions this video answers
August 2, 2023.
Cary Felt — founding product manager at Weights & Biases.
Chapters
Hi, I'm here with Cary Felt, founding product Manager at Weights and Biases and instructor for the short course Evaluating and Debugging Generative AI.
Hey Andrew, thanks for having me.
Machine learning development is an iterative process. You train a model, then maybe decide to tune a parameter, then retrain the model, and then next decide to expand the training set and so on. I used to keep track of all the different models I was training in a text file or sometimes in a spreadsheet, and after a while it gets messy and you easily forget. You go, wait, what dataset did I use last week and what exactly did I do that resulted in that improvement to the model? Accuracy, weights and Biases has become a bit of an industry standard for machine learning experiment tracking. In this short course, Carrie will teach you how to use these tools for the development of generative AI models. Compared to supervised learning, generative AI adds an extra layer of complexity because the model outputs are harder to evaluate. Carrie is an expert on these issues and has focused the last five years of her career on the process of systematically evaluating and debugging machine learning models. So it's great to have you teach this. So can you share with learners what they'll learn in this course?
Absolutely. You'll learn to monitor and debug experiments to quickly iterate and get a validated model to deploy to production. You'll trace, debug and evaluate generative models and visually compare outputs as you iterate. By the end of this course, you'll have a solid understanding of best practices for reproducible model development. Go from hacking on a notebook by yourself to validating and sharing models collaboratively with your team.
So I find that having a systematic workflow for developing machine learning projects can make you much more productive and help you get to that working model or get to that breakthrough result much faster. I hope you take and enjoy this course.
