LoRA fine-tunes AI models by adding small adjustment matrices instead of changing the original weights. Using Whisper speech recognition adapted to Kalengin as an example, the speaker shows how low-rank patterns can shift outputs while keeping the base model frozen, and why you need to test for regressions on old tasks.
Chapters
You know when you go to get your eyes checked? They ask you to cover one eye and read the letters on the wall. The big ones might be easy, but the smaller ones get harder. Then you look through that machine with the different lenses. They switch between them and ask, is this clearer? Or is this clearer? Your answers help them work out which lenses you need. They haven't changed your eyes. They found a pair of glasses that helps you see better. Keep that picture in mind because it gives us a way to understand something called LoRA. I used LoRA when adapting Whisper, an AI model that turns speech into writing, to work with Kalengin. We already had a model. We wanted it to get better at a particular job. So we gave it recordings along with the words people actually said. That is fine-tuning. Taking an existing model and giving it more focused practice. Those examples can teach it something new or improve something it already does. We don't always know everything it saw during its original training. What matters is choosing good examples for the job we want it to do. Inside a model are lots of numbers learned during training. Think of them as settings that affect its answers. We could let training change all of them. LoRA takes a different approach. The original settings stay fixed and we train a smaller set of adjustments, like finding the right lenses instead of changing the eyes. Now, this is where we leave the glasses example. In the actual model, these adjustments sit inside certain parts of the model. The same input takes 2 paths. The original path and this small adjustment path. Their results are added together. How can a few numbers make a bigger adjustment? Watch this. 4 numbers across and 4 down. Take 2 from here and 3 from here. Multiply them. And you get 6. That fills this square. Do the same thing for the other squares. 8 numbers. Now— Describe 16 adjustments. We learned those 8 numbers and they produced the larger pattern. Now, if I change this top number from 3 to 4, which squares will change? This whole column. Every square in it depends on that one number. That connection is the trade-off. These changes are linked. We can't set every square however we want with just this one pattern. Adding more patterns gives us more flexibility, but also more numbers to learn. The number of patterns is what we choose when we set the rank. Low-rank adaptation. That's where the name LoRA comes from. We save the small adjustments, but we still need the matching original model to use them. There's another part we can't skip. Getting better at the new job doesn't mean the model will stay just as good at everything else. Think about practicing only your new songs and never rehearsing the older ones you still perform. One way to help is to mix some examples of the older skills into training too. We call that rehearsal. We might not have the model's original training data. We can still choose examples that represent the skills we want to keep. There isn't one perfect mix for every model, And rehearsal doesn't guarantee nothing gets worse. So we test. We keep some examples out of this fine-tuning and compare the original model with the adapted version on them. We check the new job and the older skills that still matter. If something it used to do well now gets worse, that's a regression. Keeping the original settings fixed, Doesn't prevent that. The adjustment changes the, the answers. Back to the glasses, finding a prescription is one step. Checking that it helps with what you need to see is another. With LoRA, we learn a smaller adjustment, then check the result. And if our needs change, we may need to adjust and test again.
