Predibase Office Hours

All right, see there's one question there already. Benefits of using Lorax's approach to adapter switching compared to other libraries like VLM. I'm not familiar with DJL, but I guess I'll have to go take a look at that one. So I'd say that Lorax's big differentiator from something like VLM or Tensor TLM on adapter switching comes in, in a couple of different ways. One is that we really optimize for this particular use case in terms of performance in a big way. So when we've done benchmarks, and I shared, you know, a little bit of this in my presentation last time, but we do benchmark a bit better than VLM in terms of single LoRA performance. And that mostly comes down to Couple of things. One is that we use a kernel called SGMV, segmented gathered matrix vector multiplication, whereas they, for the pre-fill section of the compute, whereas they only use a kernel called BGMV. Basically the way to just think about is that for pre-fill, which tends to be a more compute-bound operation, the way that we do that, that particular kernel happens to be optimized for that type of, uh, computation profile, whereas the other kernel tends to be better during the decode step. So we use both kernels, um, depending on what step you're in of the, of the, um, generation process, whereas VLM only uses one. So that's one advantage we have. Um, but other things I think are a little bit more user-facing would be dynamic adapter loading. I think that's one that's— I think we're the only library that does that one, interestingly enough. Most libraries require you to kind of pre-specify what adapters you want to load in and then able to process it. And that's mostly just kind of to simplify things on their side. They don't want to have to think about like carving out separate memory pools for like adapters and then thinking about how to, you know, swap, you know, between like host memory and GPU memory and things like that. So we put a lot of effort into the scheduler component of Lorax where we kind of intelligently figure out as requests are coming in, which ones should be resident on GPU and be batched together versus which ones should be sitting on host memory, and then kind of figure out like how to trade off between latency and throughput to figure out the kind of optimal plan for when to move things to GPU, process them, and then when to offload them. And then last, I would say, you know, we do support LoRA switching, which is the most popular type of adapter, but, um, we are also leaned into other types of adapters as well. Um, so in particular, the speculative decoding work that I mentioned last time and the way that you can combine that with LoRA, that's something that's pretty unique to us at the moment. Um, and, uh, we're also looking into other types of adapters, uh, beyond LoRA. I think right now the one that might be most interesting to me is, uh, ReFT, or like representation fine-tuning, uh, is a very good candidate. There's also DORA, a few other variants like that that we're thinking about adding. But in general, we want to be a place that is very, you know, leaned into using different adapters other than just LoRA. And, you know, one example of that as well is we're adding support for embedding models and being able to think about training embedding models specific or training adapters specific to embedding models and being able to support that in the platform as well. All right, so long-winded explanation, but that bought me enough time to get to the next question. So based on what you have seen from customers, can you tell us more about the quantity of data needed for quality fine-tunes, if there are good solutions for data sourcing generation if you don't have enough to meet those needs? Perhaps it varies case by case, but curious if there are generalizations you can make. Definitely would love to have Arnab speak to this one a bit more. He's been doing a lot of exploration on this. I would say kind of the high-level summary from my perspective is that we typically ask that people generate at least on the order of like single-digit thousands of samples in most cases. I don't— it's definitely something that varies case by case, but I think that's generally a good starting point. And then for data sourcing, we've gotten a lot of bang for our buck from synthetic data generation. And so I definitely think that that's something that people should be exploring more as they're starting to look to do fine-tuning. But yeah, Arnav, anything in particular you wanted to mention on this point?

Yeah, two things. One, that usually like data volume is less important if your data quality is very high. Like the problem that a lot of people have is that they bring in these like, you know, 200,000-row datasets, but not all those not a lot of that is very useful. So your model doesn't effectively learn. So yeah, high quality data is better than high volume data. The second is that you can actually start on the order of hundreds of examples. But the way to think about it is that if you have a smaller dataset, you want to use, you want to start with a bigger base model. So say like Llama 370 billion, because those tend to do much better with smaller datasets. Obviously, that is a bit cost prohibitive, both on training and on inference relative to smaller models. So you want to eventually collect enough data to try and do smaller models, which benefit from a larger dataset. And just as a sort of like a data point to sort of support some of the things I'm saying, there's a paper from Meta probably sometime last year called Less Is More for Alignment, where they just took 2,000 samples, trained on Llama 2, 70 billion. The older generation, and it showed like really, really strong performance. So that's the way I would think about it. And certainly synthetic data increases performance pretty substantially, particularly if you have a small dataset. So you see less gains as you have like you're in the hundreds of thousands, but when you're on the hundreds or the low thousands, doubling your dataset will lead to a significant increase in performance.

Right.

Um, how do y'all usually work with people with, uh, datasets?

Like, do you find that it's hard for people to grok this dataset creation thing?

Oftentimes, yes. I would say the biggest challenge we run into with dataset creation is a user that comes in— I think it's— I think this is maybe the most classic problem that people have with like, uh, trying to go from using a closed-source API like OpenAI to fine-tuning, is you initially expect that you just want to take something like OpenAI, apply it to an open-source model, and get it to just understand the facts of what you're trying to tell it, right? So you'll say like, you know, the you know, my favorite color is purple, and then like expect it to learn that so that next time you ask it what's your favorite color, it'll say purple, right? Um, and so they have these QA pairs that they formulate and then try to get it to just learn to understand those facts. And that obviously does not work. And so that's, uh, particularly well anyway. And so that's, I think, where we have the biggest challenge on dataset prep is say, okay, like maybe we need to reformulate this as like a RAG problem or something and think about how we can you know, take your corpus of data, index it, and then, you know, train an embedder model or train a generator model. Um, but that's, I think, where we run into the biggest challenges on dataset creation. Arnab, I'm curious if there are any other examples that you can think of where we've worked with folks on certainly helping them, like, bootstrap more data with, like, synthetic data has been one we've done recently, but curious if there are others.

Uh, So what do you point people to when you try to help them with data? I mean, I know you folks are running a product, not a consulting company necessarily. So is there things that you try to give people?

Let's see. I wouldn't say that we necessarily point people to specific tools for data prep. There are consulting companies that some of our customers have worked with that will do some of this work for you, where they'll— you know, say, take a corpus, you know, help you translate, turn into a fine-tuned dataset, or, you know, just give you data for a particular task. So we definitely are familiar with some people that worked with that. We've recently been doing a little bit of collaboration with companies like Gretel that do some synthetic data as a service, and that's, I think, generally been like a pretty good relationship that we're hoping to lean more into. So shout out to, to those folks. But I'd say it's still pretty, pretty early days on that front. Most of the time we do expect that users have some well-formulated data already that they're working with. And it's definitely, I think, the complexity of that varies from use case to use case. But I'd say for us, we tend to like the use cases that are maybe closer to traditional supervised ML, like classification, NER, summarization. And in those cases, the dataset formulation is a bit simpler, I'd say, because it's much more straightforward, like input-output that you expect. And so in those cases, we don't have to do as much kind of consulting work on, on the data prep side. However, eval can still be a little bit complicated in some of those cases, but that's another can of worms.

Do you What would your guidance be for someone who's trying to figure out, do they want to use a platform like Predabase, uh, or paid platform as a service like Predabase, versus, um, serving Lorax on some hyperscaler, just using the hyperscaler compute in Lorax?

Yeah, I mean, um, I would say that it's really a cost-benefit analysis, is like, if your company wants to make ML infrastructure a core competency or differentiator from your competitors, then I can understand why someone will want to self-host if they think they can bring something to the table that is unique in that way that, you know, their competitors wouldn't be able to do very easily. And so they're willing to like staff a team of people whose full-time job is to do ML infra. However, you know, I think they just need to be, I, I think some people have this overly optimistic mentality that like, oh, I can have 2 engineers do this for 3 months and build an ML platform. And then, you know, great. Job done. I think that in reality, it's like, okay, you're going to be paying like at least 5 people full-time salaries for the rest of forever to maintain and keep this up to date. And so then I think the argument is much more in terms of this build versus buy thing is like, you know, is it worth it for you to do that? Like, do you think that, you know, your company is going to do enough that's differentiated on this front to justify that cost? or if it's what I think is the case for a lot of companies where, you know, their core competency is something else like a great B2C product or, you know, really good model that they want to be able to produce, but not worry about the infra component, then I think it's a much more clear like, hey, the cost that you're paying to say Predabase or a company like us to do all this work for you, you know, is just stuff that you could be better or resources that could be better spent doing something else, right? If you're, you know, just pay, pay a company like us the money money and then no longer have to worry about it, that's like money well spent if it frees up your resources to focus on other things.

Cool. It seems like there probably should be a lot, vastly more companies that want Predabase than want to self-host Lorax. Are you— when you, when you first released Lorax, what was your expectation of how many organizations would be using it or contributing to it, or, um, yeah, like engaging with it outside of Predabase?

Um, I think it was hard to say. I mean, certainly I had some experience with past open source projects that I worked on, and, um, you know, I think we saw two different kind of things play out with the two other projects that I worked on in the past. So one was Horovod, and I think that one had quite a lot of buy-in from like big companies like Amazon and NVIDIA contributing to it and using it. And then, and it was primarily companies that were building their own ML platforms internally that wanted to, to use it, right? Mm-hmm. Conversely, there was Ludwig, which was much more of like an end user tool. And we had lots of people that were using it and, and trying it out, but not as many people contributing back because of the fact that it was more of like a low-code abstraction. And so You know, it was, it was something where people, the type of profile of person who would use it was not necessarily the type of profile of person who would be able to contribute back to it. And I think with Lorax, it was probably closer to a Horovod type project in that sense, where a lot of people who use it tend to be more technical. They are oftentimes wanting to build infrastructure for their companies internally. And so A lot of our best contributors are people who are, you know, self-hosting, and, you know, I would love to ideally make them credit-based customers, but, you know, love the fact that they're just, you know, they're really dogfooding the product and helping to make it better as well. And I think that's a, you know, totally healthy place for an open source project to be, or for a kind of an open core company to be. But, and I'd say that probably aligned mostly with my expectations that we'd probably have more people wanting to use this infra rather than thinking about it as, you know, purely end-user-facing tool.

We got a question here with 3 upvotes. Uh, can you share some future plans for your excellent fine-tuning index that's currently averaging performance over 31 diverse tasks?

Yeah, so definitely a big shout out to Tim from our team for his work on that, as well as Justin. Both, both like really good. They were also the ones who did the excellent LoRaWAN paper that we published recently. So I'd say that with the fine-tuning index, we started it as kind of a static artifact. Essentially, we wanted to have a nice visual for people to see, you know, kind of the breakdown of everything. But, you know, we've definitely had a lot of syncs internally of saying, you know, we really need this to be something that is a living, you know, artifact that can change over time as new models get released, like seemingly every day, and be able to have it a little bit more like the LM Sys leaderboard or something like that in terms of not necessarily like get to that level of ubiquity, but at least get to that level of freshness where, you know, it's constantly being updated. There are ways for people to kind of submit their own candidates to run on the test, maybe add more datasets. But definitely in the short term, what we're doing is we're adding some tooling on our side that will make it a lot easier to kind of one-click, try out a new model, add it to the index. and then be able to make the UI component of it live in the sense that it goes directly to a database somewhere and then pulls down the data instead of having it as static artifacts. So that's the short-term goal is like, even though it won't be fully self-service, it'll be kind of more like just turning a crank, like try this model, try this model. Um, and then eventually we want to get to a point where it's more like, yeah, the Hugging Face, the LM Sys leaderboards where It's very open and, you know, anyone can add a submission. And, um, I'd say the one thing that I'm definitely very keen on in the short term as well is moving to ELO scores for the models. 'Cause I think we have a few cases where some models are ranked, um, a little bit worse than they might otherwise be because of like some outliers in the dataset. And so that's another thing that we're hoping to, to shore up. But, uh, yeah, those are a few things coming down the pipe for that.

All right, uh, this next question from Michael Angelo is a little bit long. You want to read it?

Uh, sure. So I have a large quantity of freeform and badly ran tags in the order of 1 to 2 million per month. I would need to ask 5 to 10 binary yes/no questions to each of the snippets. I know that GPT-4 performs really well, but it's, of course, cost prohibitive. What would be the best setup in this case? One adapter with 5 label outputs, 5 different adapters, something else? Also, my data is quite sensitive and based in EU. Do you have EU-based GDPR-compliant servers? Yeah, so I think just kind of taking this one step at a time, this is definitely— I think we definitely see this a lot. In terms of users wanting GPT-4-level performance but at a price point that is, you know, much more reasonable. And so definitely I think a lot of people are doing this kind of GPT-4 distillation type workload, which I think is, you know, this sounds like a classic example of that where, you know, you want to use GPT-4 as a source of truth, a source of label data. Obviously the there's like an open question around like their terms of service and things like that. But I think if you're doing it for internal models, uh, doesn't seem to be something that a lot of people are balking at. So that'd be the first suggestion for structuring it. And then in terms of, um, how to think about in terms of, uh, multiple adapters or one adapter with different labels, um, my general suggestion there is that I think the more narrow the task, the better for performance in most cases. So if you could restrict it to where you essentially know the request coming in, like if it needs to use this particular adapter and then can reduce it to a binary yes/no, that's a lot easier than trying to train a model that knows how to handle all these different cases. But I think it depends a little bit. So the thing that would be the biggest deciding point here for me would be, Do you actually have enough information to know a priori when the request comes in which adapter you would need to send it to, or is it the case that, um, you don't know that and you'd have to go ahead and just run it on like all 5 adapters simultaneously? If it's the latter, you're probably better off trying to train a multi-class classification, um, multi-label classification model. But if it's the former where you do know, you know, which adapter you need to use, then I would always recommend just going with that approach. So it's a little bit task dependent. I think I, I don't have enough detail there to know for sure. And then of course there's also things with routing architectures that you can do theoretically to, you know, figure out which adapter you should use and things like that. And then in terms of GDPR compliance, Yeah, so we've definitely been working with a lot of partners in kind of a POC capacity who have data centers in Europe. And so if that's something that is important to you, definitely reach out and we can talk about options there. So we have been working with some EU-specific cloud providers that are GDPR compliant who can run the PradaBase stack in their environments. Happy to discuss that use case in more detail. Just travis@predabase.com or support@predabase.com are both, both good emails to use.

All right, we got a couple in here that are related to Apple. Apple is swapping LoRAs on device. Could Lorax work on this edge device level? And then also, can you speak more about look-ahead LoRAs and how they might work in practice? Uh, and then would you need to fine-tune each time with Speclab decoding setup?

Yeah, so the on-device use case is definitely one that, um, we've been talking about for a while, even predating the Apple announcement, because I think it's such a good application of something like Lorax. You know, you have small language models, you want to be able to get them to do a variety of different tasks, but you can't serve a 500 billion parameter model on your phone. So being able to— and the thing I like to say is, like, you know, MOE models are very interesting and useful because they take advantage of sparsity of the task. It's like, for any particular query, you don't need to necessarily activate all of the weights to answer it, right? And I think that this kind of mixture of LoRAs thing is like taking that concept to an extreme, right? It's like, for any given task, there's this huge space of potentially billions of adapters with weights that you could use, but you really only need, like, one of them. To answer the question. So, um, definitely I think this— the future is probably going to be sparse activation one way or another, whether it's like built into the architecture through like, you know, systems like what Apple is doing with a semantic index and then using an appropriate LoRA based on the question. Um, but regardless, yeah, I mean, I do think that, um, LoRA running on edge devices is something that I would love to explore. Uh, we have talked in very preliminary sense with some different people who are interested in this use case, uh, for different types of hardware. So there's definitely, you know, Apple devices, but other companies out there have hardware that runs on embedded, like Qualcomm, for example, has some new chips, uh, designed to run on device for AI. Um, so I definitely think it's something that, uh, would be very appealing. I think the challenge right now for us is that, uh, we've optimized Lorax very specifically for running on NVIDIA GPUs, and NVIDIA GPUs tend not to be, uh, run on phones, uh, or other embedded devices. Um, so things like flash attention, things like, um, paged attention, things like the SGMB kernels that we use, all written in CUDA, you know, so that makes it difficult to make it portable. And some of those techniques might not apply as well to different, you know, ASICs that you might run on a phone to accelerate neural network computation. So that's something to bear in mind is like a lot of the optimizations we make might not necessarily apply. So it'd be a big rewrite of a lot of things, but at least at conceptual level, I think it's like such a good fit that I would want to find a way to do it. And we were actually talking about this. Not too long ago about this for this very reason. It's like, it just seems like such a good fit. A couple other things that were asked about here: speak more about lookahead LoRAs and how they might work in practice. Yeah, I mean, I think that the main idea behind this, like, fine-tuning for inference speed concept— I think we're playing around with different ways to market the terminology right now. The one I like the most is probably like low-rank acceleration. It's like, can you, you know, fine-tune to make things faster? Um, but my, my expectation is that, uh, they will work well in practice, um, because, um, and I think this bears out from the data that we presented and that we're working on, you know, putting together for kind of like a larger blog post/paper, um, is that Spectral Decoding in general just works a lot better when it's like a very narrow task, the same way LoRAs work very well for very narrow tasks. Because you, you know, if you're doing a summarization task and your, your speculative decoding model was trained specifically on summarization tasks, you know, it'll do a very good job because it's a very narrow thing. It's like, you know, you're mostly looking at the text that's already there and like trying to predict some, you know, reduced form of that. Or same for like structured generation. It's like if the model learns the JSON format, it's going to be very good at predicting that, right? Um, so I think it'll work very well in practice where you have very narrow task definition, and the more narrow the task definition, the better it will work. So I think if the trend continues to be towards using SLMs, small language models, for very narrow tasks with like LoRAs that are fine-tuned to those specific tasks, then I think things like speculative decoding will be a natural fit. Um, a couple other reasons why I believe this will be true. If you look at the newer generation of NVIDIA chips as well, like if you look at the— compare the A100 to the L40s, like going from Ampere to, to Lovelace, um, it's really interesting to note that the memory bandwidth on the L40s is lower than the A100, but the flops is higher. And that's kind of interesting for speculative decoding because, uh, speculative decoding works best by essentially like increases the compute overhead, right? And so in a case where you're, uh, have more excess compute capacity, speculative decoding is going to give you more bang for your buck. And so if the trend continues to be towards, you know, there that NVIDIA is reaching some limitations inherently on like how much they can move the needle on low-cost memory bandwidth increases, but increasing the flops with different generation cycles, and that would be another reason why something like speculative decoding could be a natural fit over time. So yeah, some just initial thoughts on why I think it's, it's very practical. I think the biggest challenge that needs to be solved right now with it is just how to make it work well at high QPS. But we have some kind of early preliminary results that are looking promising there as well. Um, and then would you need to fine-tune each time with a speculative decoding setup? So the way that we think about it is that, um, like fine-tuning the speculative decoding part would just be something that happens when you fine-tune the LoRA for the task, um, and that users hopefully shouldn't even have to think about it. Like they just think about fine-tuning the same way they do today, or it's like here's my dataset fine-tune. Oh, and it just happens to be 2 to 3 times faster. So that's our goal. And, uh, you know, with the— with what's in the platform today, that's essentially what you get as well, the way that Arnav has architected it. So, um, yeah, should work very well.

Okay, we've got next question is when you say synthetic data, can you effectively bootstrap by using a large model with a small number of input data for fine-tuning, then use outputs of that to generate more value than using a small model?

Yeah, absolutely. We definitely have seen this work pretty well, surprisingly well in practice. Yeah, Arnav, I know you've been doing a lot in this space recently if you wanted to say more.

Yeah, so certainly you can use like a large language model, small number of input data, fine-tune that model to generate more, and it works really well. We've seen pretty much, I'd say for every 2x increase in data, about a 5% lift in performance. So pretty significant. The one call that I'll make here though is that Where I think this would be most effective is if you use the same model family. So if you're using a Llama 370 billion as like the large language model initially, then you might want to distill down to Llama 378 billion. But I'm not sure how well it works cross-families distillation-wise just because of the distribution of words being produced. And you can also do this slightly differently where you take GPT-3.5, GPT-4, take your small number of input, take your small input data and basically ask it to generate more synthetic samples and then use that to just fine-tune a smaller model.

Yeah.

You know, that sort of sparks something that I haven't tried but makes me curious about it. In conventional pre-LLM machine learning, a lot of the models worked well by some sort of, by like diversity or some sort of ensembling, even like random forest. It makes me wonder whether when you generate synthetic data, there's some value in generating the synthetic data using a bunch of different model families so that they don't like zero in on just one particular aspect of of the raw data that they were— that you would give them, but instead they have some diversity. And that all could be learned when you, uh, when you— it's not really distilling as much anymore, but like semi-distill, uh, at the fine-tuning stage.

Yeah, absolutely. Actually, are you familiar with like the data programming paper?

I'm not.

Okay, so, uh, there was, um, some work that came out of Stanford, um, from— there's a company called Snorkel.

Oh, okay.

Weak labeling, right? And it's essentially this idea is like you have a bunch of like, you know, weak experts that label with these things called weak or soft labels. And then you have a really smart way of ensembling their labels together to kind of figure out like, what is the actual expected label given all these like weak labelers together that are making a prediction? And so it's kind of like the same reason why model distillation works is that you're not distilling on the final label, you're distilling on the logits and the probabilities. And so you get that, you capture the nuance of like, oh, everyone agrees that this is the right answer versus there's some disagreement about whether this is the right answer or not, right?

Yep.

So that actually, it does work quite well in practice. Let me just ping the data programming paper in chat here for everyone. Let's see. I think I can only do hosts and panelists, but, uh, we can—

I will put it in the, uh, Discord.

Oh, perfect. Okay. Yeah, definitely worth checking that out. We actually ran some tests on this last year, um, for a very similar situation where we wanted to see— this was like an early experimentation with LLMs to see if we could have like a whole bunch of different experts generate a prediction, zero-shot, and then fine-tune something on the ensemble of all their predictions. And we did find that actually outperformed any individual expert by fine-tuning on kind of the ensemble of all of their predictions. So definitely bared out. And I know that Snorkel published something about this as well. Um, distillation last year. Let me see if I can find that. Yeah, my— I'll definitely, uh, send it in the, in the, in the Discord if I find it. But, uh, yeah, I think it's a really cool idea for sure.

All right, that's I'm going to read the next question. We've got this one from Stefan. Impressive work on the fine-tuning index. How do you see the potential for fine-tuning adapters to replace GPT-4-level models in open-ended Q&A over company internal documentation? Super popular use case to talk about. Is it worth it to try fine-tuning for such a complex use case? And how would you approach it if you already have a GPT-4+ RAG system in production?

Yeah, so there are different ways to think about this. I think that the long-term promise that people want from fine-tuning is that fine-tuning could replace RAG entirely by just fine-tuning over a corpus. You do domain adaptation, suddenly the model knows everything about your data and can just answer questions. That reality, I would say, is not here today, unfortunately, but I still hold out hope that someday that That'll be true. And I think that once that is true, then hopefully companies like Predabase will 10x our values. But I think in the meantime, what we do see that works well is actually trying to tackle the different components of the RAG system through fine-tuning and trying to provide some level of lift at each of those aspects. So for example, typical RAG pipeline today might have an embedder, a re-ranker, and a generator model, right? Each of those individual components can be fine-tuned. From all the literature, it bears out that fine-tuning does improve performance on those. Indeed, there's some papers that demonstrate that you can jointly fine-tune these components as opposed to thinking of them as entirely independent. You can fine-tune them jointly at the same time and get the best performance of all. I believe Arnav remembers the name of the paper on that one.

It was called RAIDIT, or Dual Instructural Fine-Tuning, where you fine-tune the embedder and the LLM together.

Yeah, so definitely I think that there's some evidence to suggest that fine-tuning for RAG is a very real thing. I think that last year, you know, was a bit early for this. We were kind of wanting to explore it and not a lot of people were ready for it. But now I think we're starting to reach a point where RAG systems are going to production. People are finding out that like getting something wrong is a very big deal in production. I think right now the biggest ROI comes from better and better models, so doing better retrieval of documents. But there's room to improve on every aspect, and so I definitely think fine-tuning has a big role to play in this.

On the— so I think there was two parts. So one, the original question was like, can fine-tuning replace RAG? And the second was, within RAG, can you fine-tune the embedding and re-ranker models? On the first part, one of the things I think many people overlook is even if fine-tuning worked super well for this, if you've got a set of documents that's changing over time or that you have documents that get added, updating your fine-tuning on a regular basis is probably much more difficult or much more costly than adding new documents to your, to your database, which is what you do with RAG. And then obviously with RAG, you do a lot of metadata filtering So you can say like, I want this document, but only one produced in the last year. Um, I think getting these details right in fine-tuning, even if fine-tuning otherwise could remember the right content, um, I think will always be a difficult problem.

For sure, for sure. Yeah, I, I think that the freshness aspect is like another completely orthogonal angle that needs to be solved. It's like, yeah, they keep fine-tunes up to date with new information. There are some really interesting ideas that have been proposed out there, like being able to stack LoRAs together, where it's like you train a LoRA and then you get new data, you train another LoRA on top of it and compose them in some way. But I mean, this is combining two different things that are both like, you know, very early. So I would say that, you know, getting to a point where both are ready for production is gonna be quite a while from now.

Yep.

But the good news is fine-tuning a better model or fine-tuning a re-ranker, those are things you can do today. And like, people have demonstrated provide very significant benefits on cost and quality, like, in the, in the present. So definitely worth doing if you have a RAG application.

We, um, just got a question which I think is a particularly interesting one in the context of distillation. Um, is catastrophic forgetting inevitable even when only using a single LoRA?

Definitely. I think we found that you can still get catastrophic forgetting or like collapse of the model even when using a LoRA. Like if, um, you know, whatever comes out of the LoRA results in, you know, you can just kind of as like an extreme example, the LoRA produces, causes them all to produce NaNs or something like that, or zeros, then, you know, it'll catastrophically forget and collapse and the model will be garbage. So it's definitely still possible. But that said, oftentimes we say that catastrophic forgetting is less of an issue with LoRA than it is with full fine-tuning, and that's certainly true. And this was kind of borne out in that Databricks paper. It was like LoRA, you know, remembers less, forgets less. I think that's what it's called.

Yeah, it learns less and forgets less.

Yeah, yeah. And I think that that is consistent with what most of us know as well is like LoRA, you don't do LoRA fine-tuning typically to, similar to what we said before, teach new knowledge or do domain adaptation. Typically, LoRA is best suited to getting the model to structure its outputs in a certain way and do better at more narrow tasks like prediction or NER or things like that. Um, but, uh, that being said, um, that is one of the benefits, is that you also don't have to worry as much about these sort of collapse or catastrophic forgetting scenarios. So while it can still happen, we do find that it's very rare when the output of the fine-tuning is something that is like complete garbage, or, you know, like forgets everything that it knew, right? Like, we see much more consistent— consistently seeing some amount of lift with LoRA fine-tuning for any given task. I think that's like a pretty consistent thing. And then it's just a question of like, do you see as much lift as full fine-tuning for whatever task you're trying to do? And certainly for the tasks that we think are most suitable for production, like NER, summarization, things like that, like LoRA tends to do quite well on those.

Uh, we've had a question that's been, uh, sitting for a while. Probably has a short answer. Does Predabase have any plan to support the new Qwen2 models released, uh, recently by Alibaba? They've got 1.5 and 0.5 billion parameter models which can be used on, uh, edge devices.

Yeah, yeah, the good news there is that, uh, actually Tim, the person who did the fine-tuning indexes, uh, just wrapped up, uh, doing the validation to make sure that Qwen2 fine-tuning and serving worked end-to-end on the platform. So The plan is to go live with that this week with serverless deployments for that. And so you'll be able to fine-tune, serve, test it out, and then do, you know, kind of a production deployment from there if you're happy with the results. So that all should be working end-to-end this week, ideally.

Yeah, we've got a question from Andrew. I've sort of, it's got, it's got an upvote. I think you're about to go into a little bit of just describing the matrix factorization of a LoRA. So the question is, if you have multiple LoRAs, do they update the same subset of model parameters or are they random? You can take that and then after that maybe we can save the second part of the question after we've talked through, yeah, this idea of how do you have something that's in some ways less flexible Uh, which is the, the idea of like a low-rank approximation.

Yeah. Um, so for the first question of if you have multiple LoRAs, they have— I, I assume the question is they have to update the same set of model parameters. So if you have like 3 different task-specific LoRAs that you load in and you want to process them in a single batch, do they all have to be targeting the same layers? That's kind of how I interpret the question. Um, And so the answer there is luckily no. And the reason is because we actually, you know, so I mentioned that the kernels that we're using for the different segmented matrix multiplications came from the Punica project that was done by University of Washington, but we did make some modifications to those kernels. One of the modifications we made was to allow you to specify like basically do a sparse version of that segmented MatMul where you can say that certain segments, you know, meaning certain rows in the batch, can be excluded. And so what that means is that if you have a LoRA that targets, say, QKV, and then you have another LoRA that only targets Q, you can still batch them together at the same time. And then for the, the rows that don't need the K and V, or that don't target the K and V, you just basically can mask them out essentially, or like, you know, just skip the computation there. So even if you have this kind of heterogeneous setup where each LoRA targets different layers, they can still all be batched together at the same time. So that works pretty nicely. And then, can you specify which layer the LoRA is applied to in Lorax? Yes, so that's done in the PEFT config. So, you know, we do accept, accept LoRAs that are fine-tuned using Hugging Face's PEFT library. And in that, in the config, there's a section called target_modules that tells you which layers the LoRA targets. And so we honor that when we load the LoRA and then say, okay, you know, make sure that like for this layer, like this particular weight gets applied, etc. And then let's see, what was the other— was there another question you said that followed up from that?

Nope, I think you got them both. I, I interpreted the first question differently than you, but your, your interpretation is probably the right one.

Okay, good, great.

Um, what kind of LoRA depths have you experimented training with? How does the parameter count scale with the amount of data you're training with?

Yeah, I, I'm guessing by depth here we mean rank, like the LoRA rank we're using. Okay. Yeah, so we've experimented with anything from like 8 to 128. I think Arnav, you might have some good insight here. I know that we've kind of settled on 16 as the default, but yeah, do you wanna speak to that?

Yeah, absolutely. So we've tried everything from 8 to 128. I've played around with this quite a lot myself to see just how much performance we can squeeze. What I found is that 8 is a good starting point, which is the default that PEF ships with, but 16 is, more or less where we've seen really strong performance and sort of the rest of the industry has converged on with LoRA fine-tuning. As you increase ranks, like the model performance does get better, but what I've personally seen is after rank 64, there's a bit of a collapse where the performance starts to get worse again. And I haven't quite figured out why that is the case. One formulation on this could be that there's a, this parameter called LoRA alpha, which is used to sort of decide how much of the base model's weights contribute to the overall concatenation with the LoRA weights. And traditionally, the formula is that you do, like, LoRA alpha is 2 times the rank. My sense is that if you were to reduce that maybe with higher ranks, you'd probably still be able to get good performance. So, but that's something I haven't had the chance to try. So up till 64 feels good. Beyond that seems to get worse on quality. And in terms of parameter count, it doesn't actually scale so much with the data you're training with, but it scales more with the rank of the LoRA. So for every increase, like 2x increase in rank, you're increasing the number of trainable parameters also by 2x. And you can keep increasing this till pretty much the length, I guess, of the linear layer, but at that point you're significantly increasing your training cost.

Yeah, I'd say— oh, go ahead.

Yeah, I think one, just one other thing to add on, I mean, definitely totally agree with all everything Arnav said there. Um, I think definitely another thing that we've noticed is worth playing around with is, you know, not necessarily just the rank but like which layers you target. Um, and for some applications, you know, targeting the default of like QV or QKV is pretty normal, a lot of applications we've seen, particularly if it's generative use cases, targeting all linear layers or even targeting the embedding in the LM head tend to work well and give really good performance benefits. So definitely increasingly, I think we're, we're kind of pushing users to try playing around with like which layers they target in particular when doing fine-tuning.

I've sometimes had an intuition, or always had an intuition, that, um, alpha should scale not linearly but with, um, the amount of data you have, so that if you're gonna have very little data and what you're learning is probably going to be quite noisy, then you should try and downweight it. And if you have a lot of data and it's high quality, then you should rely on what you're learning more heavily. Does that— is that your experience, or is that your intuition as well?

Yeah, that sounds right to me. Yeah, like most people treat this as like static, either R by 2, or sorry, alpha by rank or alpha by the square root of rank. But I, yeah, I would agree that probably you want to upweight it if you feel pretty confident that you're not, the tricky part is that you also don't want to lose like base model knowledge, right? So you need to feel really confident about the LoRA performance. The good thing is, uh, yeah, it's pretty easy to play around with these and just get a sense for performance.

Yeah, it's interesting that it's— because it's a constant, it's essentially just an inductive bias at the beginning of training, right? So yeah, it's, it's kind of like, you know, trying to, to bootstrap, I guess, in some sense, like a sense of how much to weight it in either way. But in some ways, you think this would ideally be something learnable, right? Is like, through the learning process, determine how much we should weight, which I guess it does through the magnitude that it learns. But yeah, it feels like this is something that should ideally be solved kind of like through the learning process itself. Cool.

Uh, I have a specific requirement where I need to fine-tune a model on a corpus of books where the system needs to be able to both answer questions and be able to quote any part of the book verbatim? I've been trying to get an answer for the whole conference. So actually, we've got 2 questions. The remaining 2 questions feel very similar to me. They're both like RAG versus fine-tuning. You want to take— I think the one from Anonymous is a specific version of the general question about when do I use RAG and when do I use fine-tuning.

Yeah, I mean, I think this is definitely where you cue the, like, meme of the girl with the cereal, like, why not both? And I think that's my favorite recommendation is, like, I think that RAG is the best tool in the toolbox today if you're trying to get exact answers to kind of fact-based questions. And in particular, if you're trying to do citations, I think you kind of have to use RAG because You know, it's basically impossible to get the model to be able to give you a citation of something from its weights without hallucinating wildly, right? So definitely, I think you'd want to use RAG, particularly for the first use case. And, you know, I would say that where fine-tuning has a role to play is similar to what I said before, where if you— like, here's how I would start every RAG project, right? Is like, start with like the off-the-shelf OpenAI type stuff, you know, pre-trained and better model, pre-trained generator model, etc. Get something working, see what the results look like, find a good eval framework to determine whether or not it's at your quality bar or not. And then if it's working well and you're happy with it, start there, maybe try seeing what the costs look like. And then it's a question of like, what's my biggest problem? Is it that the costs are too high or the performance is too low? And from there, like start iterating. And in terms of the iteration process, I look and see, like, try to diagnose where the problems are, like where things are breaking down. Like, is it breaking down because the embedder is failing to retrieve the correct context? In which case, that's an opportunity to attack the problem with fine-tuning to say, let's try to figure out a better way to index or to retrieve these documents so that the correct context is always there. Conversely, if the problem is the context is there but the model is just giving the wrong answer, then there's an opportunity to fine-tune the actual LLM that does the generation at the end as well. And again, there's most likely, if you want to get the best results, there's an opportunity to kind of jointly fine-tune these things together as well. Um, but to me, it's an iterative process. Is I would not ever, you know, go right out of the gate guns blazing, like, let's just fine-tune everything. Um, I would start with baseline performance, similar to like old school data science and machine learning. Pre-LM is like start with a simple model like linear regression or something, and then slowly layer on more complexity to see like what the lift is, um, and kind of work your way towards a better and better solution through fine-tuning.

Yeah, I think we're going to take one last question before you answer this question, but I'd love to hear your answer. I should mention that In something like 3 hours, you have to look on your calendar, but in roughly 3 hours, we have a session that is specifically about fine-tuning for function calling. So love to hear Travis's take, and then we'll hear another one pretty soon. So what about function fine-tuning for function calling? Should you have a LoRA per function type or go for a multi-skill one?

Yeah, so I'll first comment on this by saying that I'm definitely not an expert in fine-tuning for function calling. But my intuition on this one, um, as kind of a user of function calling systems is pretty similar to the advice I gave previously, where it really comes down to what you know about the query at the time that you're sending it to the LLM. So if you happen to have some metadata that tells you what type of function this is, then having a LoRA that's very specifically tailored to just that function type is probably going to be the way to go, just based on that simple heuristic of the more narrow the task, the better the performance in general, the better the lift you're going to get from fine-tuning a LoRA. So if you have something in the request payload that you can key off of, like, you know, I know that this is a Python function, or I know something else about it, then you can have a LoRA for Python functions or a LoRA for, you know, something else, right? That you can, you know, narrow the request to. That's where I would— that's the level that I would break it down at, is anything that you know, like, at request time that you can take advantage of and narrow the space, the cardinality essentially, down. Like, train a LoRA, like, at that level of granularity. Yeah, not sure, Anav, if there's anything you want to add on to that.

No, that sounds— that sounds like— that sounds right.

Awesome.

All right, well, this was great. I certainly enjoyed it. I think everyone else did as well. So thanks, Olav and Travis, for the office hours and credits and for Travis for the previous help on the deployment workshop.

Awesome. Thank you all. Thanks everyone for coming today and thanks Dan and Hamel Husain for having us. Awesome.

Thanks.

Bye.