Creating, Curating, and Cleaning Data for LLMs

So the plan in this session is to kind of do a little bit of a high-level overview, focusing on this topic of creating, curating, and cleaning data. So I think this has already been discussed quite a lot in the course, and I think it's come up various points. So there's probably some stuff that we'll say that you'll already be familiar with, but the idea is to hopefully give you some ideas for how to approach building datasets for fine-tuning large language models. So I'll just quickly introduce myself and then let David introduce himself, and then we can kind of dive in. So my name's Daniel Van Streem. I work I work as a machine learning librarian at Hugging Face. I can talk more about what that means at the end of the session if anyone's interested. But my background, as the kind of name implies, is very much in libraries. So I kind of fell into machine learning and large language models in the same way probably a lot of other people did via the fast.ai course that I took a couple of years ago now. But I think the library background is kind of nice for this topic because one thing that you will do a lot in libraries is look at lots and lots of data and think about how to organize data and how to structure things in a systematic way. And I think that is a big part of working with data in the context of large language models. So I think those kind of skills can be quite useful even though they're a little bit outside of what you'd expect. For people working in this domain. And David, do you want to do a quick intro and then kind of dive in?

Sure. So I'm David. I'm doing ML and, and DevRel at Argila. And Argila is this data collaboration platform for AI engineers and domain experts, for the ones that aren't familiar with it. And yeah, I kind of started off with, with NLP and these kind of things by, by initially my master's, and then working in private intelligence, working on knowledge graphs and also, yeah, custom domain NLP models. And that's also, yeah, I think where data quality and model quality is very, very important. So yeah, that's kind of my background, and I will be covering the synthetic data part and hopefully give some pointers on that for you guys.

Okay, that sounds good. So I'll jump in then. So basically, the kind of structure of this presentation is to start from like the ideal case in which you might already find some existing data and then work to the probably more realistic case that you're going to have to build some of your own data for fine-tuning if you're not working in a topic that's already kind of well worked on. So I'll just talk very quickly about some ways in which you might find existing data. So obviously, I'm going to pitch Hugging Face as a good place to look for datasets. And I think that the question is a little bit what kind of datasets you need to look for. So the thing with Hugging Face, there's quite a diversity of datasets that are shared on the hub, but a lot of the ones that will be trending and very visible tend to be focused on like a very kind of specific type of use case. So an example of that here is this FineWeb dataset, which probably quite a lot of you've seen do the rounds on social media. So this is a kind of dataset focused at pre-training large language models. So though it's like a very interesting dataset, it's probably not something you're going to find particularly useful unless you're actually training base models yourself. The other thing that I would say is that there are a lot of research datasets that are associated with papers that can be really interesting, but I think there's also a growing number of community-contributed datasets that are made by people doing slightly more kind of bespoke and sometimes weird stuff, but those datasets can actually be really valuable, both for using directly, but also for getting a bit of a better sense of how people are approaching building datasets. And so one of the ways I would suggest, if you're not already familiar with finding datasets on the hub, is to use this kind of tags feature. So I think that's not super well used at the moment, but there's kind of growing usage of it. And particularly for things like DPO datasets, it can be a really useful way of finding datasets that kind of match a specific format, and I'll talk a little bit more about DPO in a little while. There is also this full text search, so if people have done a kind of bad job of naming their datasets, this can often be quite useful way of finding datasets because it's looking in the full dataset cart to find things. And then once you found the dataset, ideally someone would have documented it really well and explain exactly what it's for and what the limitations are and how it was made, but that often doesn't happen. But one thing that I think is really nice with the hub is that you have this datasets viewer that gives you a kind of preview of the dataset. And I think that can be a really good way of doing this kind of vibe check on whether a dataset is going to be useful for your application or not. So there's a kind of example of what that might look like here. So there's this dataset for doing function calling. And as you can see, it's like very badly documented. There's no information needed. I'm not trying to pick out on this person, but it's just quite common that people leave that documentation till later. And one of the things that you get in this preview is some kind of metadata about the dataset itself. And in this case, you have this conversation, which is this kind of chat ML format a lot of people are familiar with, where you have basically a list of dictionaries. And one of the things that might be interesting for you to know is like how long those conversations are and what the distribution of those looks like. Depending on what kind of domain you're working in, you might expect the kind of chats to be very short, or you might expect to have longer chats, and then having a sense of what this looks like. Can be quite useful. But the other thing that you can then start to do is to dive into, okay, like, what are the actual conversations in there? So you can look at actual example rows. And one of the things I noticed with this dataset, even though some of the chats messages are quite long, some of the responses or the kind of final turns are just basically the person being like, oh, thank you. And then the model being like, oh, you're very welcome. So probably that kind of response is not actually that useful. So if a lot of the longer messages are like that, then maybe you can't kind of see this as a good dataset for those kind of long conversation, yeah, fine-tuning use cases. So I said that was probably the ideal scenario that you find something that you can either adapt or kind of directly use. But I think in practice, often you're going to have to create your own dataset. And I looked a little bit through the Discord, and I think people are like, tend to be quite creative about this already. But I think one of the things that probably is like a little bit underused still is just to adapt existing datasets for large language model fine-tuning. So there's a bunch of datasets from kind of classic NLP that can be restructured or reformatted to work with LLM fine-tuning. And particularly if you're already an organization that's kind of been using machine learning for a while, probably have some of those datasets already around that you potentially could adapt. The other part is like whether you already have some feedback from users. So there's a lot of discussion about preferences and preference datasets, and you can set about creating those very deliberately either by using human annotations or LLM judges, but quite often you will already have some indications either very direct, so people have like a thumbs up or a thumbs down, But there might be other ways in which you can kind of get at this question of like, was a user satisfied with a particular interaction? And that might be a source of preference data that you can kind of create without actually having to regather all that preference data manually. And then the kind of final one is this synthetic data, which I think can go hand in hand with these other bits, but I think it's also very powerful for kind of jump-starting the process of building these datasets. I won't kind of labor this point too much, but I think there's also often quite a lot of work to get your data into a kind of format you can actually use for large language model training. And I think there's some nice tooling for this, but often it will be geared at a kind of very specific use case. But yeah, one thing I wanted to kind of mention about that already is that I think a lot of the work that you do in prepping this data might also end up being very useful for when you get to the stage of deploying a language model in production. So I think you want to have that in mind, that the kind of format that you're gathering this data from should be quite close to what you're going to actually be be using that language model for in practice. So some of this kind of preprocessing code and work will have a lot of overlap with what you're actually going to be deploying later on. So jumping a little bit more into the kinds of datasets you need for fine-tuning. So I think in some of the earlier lessons, this was already discussed a little bit. But I think in terms of the kind of I guess the question of like what needs to be in your dataset, and I know that's a little bit vague, but I think one of the kind of considerations, which I think is slightly different when you're fine-tuning for a more specific use case, is being okay with the kind of model losing certain abilities. So often when you're kind of fine-tuning chat models, which I think is what a lot of the literature ends up being about, you want to make sure that you have a very diverse dataset and that it kind of captures all the things that users might discuss. But in practice, quite often the kind of scope of your problem or the application you're trying to develop is much more narrow. And then I think you don't have to worry about the diversity of the data in quite the same way. And you have to really think about diversity in terms of what your large language model is actually likely to get as input. And I'll talk a little bit about these common dataset genres. So unfortunately, it's a little bit the Wild West in terms of like the actual way in which datasets are structured. So there's a lot of variation in this, but often there's some kind of similarity. So for the supervised fine-tuning, you have this kind of question and answer response. And this probably sounds quite obvious, but I think for some of these fine-tuning use cases, and also when you're thinking about reinforcement learning and some of those algorithms, for a lot of people, I think they find it easier to understand, okay, what does the dataset for training using this particular algorithm look like? And then they can kind of map a little bit more at a high level what the process actually looks like and possible ways that they could get to a dataset that matches this kind of format. So very kind of briefly, I won't talk about this reinforcement learning bit too much. But if you're kind of following this literature at all, there's a lot of algorithms being published. Some are kind of iterative improvements on existing ones and some are quite different. But what I think seen over the last year or so is that a lot of these algorithms end up using very similar dataset structure. So I think that's one of the kind of nice things in a way that a lot of the existing datasets can either be kind of lightly adapted or used as they are without having to do a lot of work to reformat things. So one example of this kind of algorithm that became very popular and has been discussed earlier is this direct reference optimization. And kind of going back to this idea of some people finding this kind of expression of algorithms not that helpful, I think it can then be really useful to go back to what does the model actually expect when it's being trained using this particular approach. And this direct preference optimization, I think, is really nice because it kind of maps quite well to how you kind of want to nudge a model when you're doing fine-tuning. So you have some input, and that could be whatever your task is related to. It could be kind of natural language, it could be code, it could be a bunch of different things. And then basically you want to nudge your model more to this chosen and rejected. And the reason I kind of mention this is I think one of the nice things I've seen quite a lot in the past year or so since this algorithm became really popular is like really interesting creative ways for people to actually come up with where are these chosen and rejected. So one way you could do it obviously is to like manually write a good example and then write a bad example, but that's really time-consuming and tedious. So people have done other things. So for example, with this chosen, if you already have some kind of ground truth or gold standard data that a human has generated, often that can serve as a really good value for this chosen. And then, for example, generating this rejected response from a model that you've evaluated that does an okay job, but isn't quite there in terms of what you would like the response to look like. And then the final kind of algorithm that I'll mention in terms of datasets is this KTO algorithm. And the nice thing about this is in contrast to DPO, where you need this kind of two preference pairs of chosen and rejected, with KTO, You basically just have a response from the model and then this binary preference. So basically a thumbs up or a thumbs down. And that's something that can be quite easy to collect. So I think in general, people are quite happy to say, I don't like this, or I do like this in an existing system. But again, you might also have other kind of creative ways in which you might intuitively be able to understand, okay, well, this person, like, didn't click this link or didn't do something else in this system. So probably that was a thumbs down and they did you something, that kind of implies a thumbs up. So I think this could be a really useful approach to gathering data if you already have something in the wild that can kind of be interacted with by some kind of end user. And then the final kind of one of these algorithms in terms of datasets that I'll mention is SPIN and ORPO. So, SPIN is a kind of iterative approach. So, a lot of the kind of approaches to doing this reinforcement learning have been focused on trying to reduce the amount of data you require because that's kind of one of the big bottlenecks. And the idea with SPIN is that you basically go through these different stages with some starting data and then you kind of synthetically generate new responses and then kind of build a dataset on top of that without actually having to have such a large dataset initially. And this ORPO algorithm again is a little bit about efficiency. So the ORPO algorithm basically expects the data to be in the same format as the DPO. So you have this input and then a response that's chosen, a response that's rejected. But the difference here is that it doesn't rely on you having done this self— a supervised fine-tuning step. So you can kind of more quickly go directly from a base model to actually doing this alignment without having to kind of train in two steps, which is often what you have to do with these other approaches. So that can be both nice in terms of not having to duplicate data for doing the fine-tuning part and then the preference part, but also it means that the actual model training is a little bit more lightweight because you're not having to do that in two stages. And I'll hand over to David. Do you want me to hand over the slides to you as well?

Yeah, please. So I believe you have to stop sharing for a while and then I can— I think you need to disable it in Zoom itself. Yeah, perfect. So just to kind of get back to kind of what Daniel already highlighted is that in the end, often you don't have your perfect dataset, and eventually you actually want to work towards like the, the perfect dataset that you might have. So that's a way to actually do that is through synthetic data. And synthetic data is, yeah, data that's generated by LLMs, and that's commonly used for fine-tuning LLMs. So you can actually start your synthetic data, for example, by generating prompts from scratch, by generating completions based on prompts. And normally you basically prompt an LLM along with some initial context, for example, in order to also do rephrasing. And some of these algorithms that do rephrasing of prompts to make them more higher quality or higher complexity or more nuanced, so to say, in order to ensure a higher quality prompt or completion, are the evil complexity and the evil quality prompts. And then there's also this special kind of feedback or synthetic data, and that's actually judging, judging synthetic data, and that's to assign scores to prompts and completions or to do preference ranking or to actually provide a rationale or critique along with your initial score that kind of explains as to why it came up with why the model actually produced the score. So we actually would use a prompt template, prompting a model Okay, please provide a score for this prompt and the associated response. And can you also provide a rationale along with that? And that would be kind of the prompt template, so to say, that people apply for, for these kind of synthetic data things. So one of the earlier models that was actually generated, trained on synthetic data was the Alpaca 7B model. And what they did was actually prompt the text-davinci model along with the self-instruct research paper. So it was a prompt template that was actually taking some seed instructions, so some initial seed data, and actually was prompted to rewrite that seed data to, yeah, better instructions or more diverse instructions. So they started up with 175 instructions, then eventually ended up with 52,000 instructions, and actually did supervised fine-tuning along with the Llama 7B model that Meta initially released. So that's, I would say, amazing. You can actually use LLM data to train LLMs and then everything is solved. But as you might expect, that's not really the case. So if you actually look into their report about synthetic data and how, report about the synthetic data and the self-instruction and how the model actually performed, you can see that there's a lot of hallucinations, toxicity, and stereotypes within the model. Might be due to the fact that they actually used the meta model, which wasn't trained optimally. Might be due to the fact that they actually used the DextreVinci model OpenAI, which might contain some bias. But in the end, it, yeah, would have probably been better to actually look at the data and see what's in there. So one of the examples from, from a completionist is like, what's the optimal seed? Why, why is 42 the optimal seed? And the model actually starts hallucinating that, that 42 is the optimal seed and that it can be used for, for any neural network training. So another example would be maybe using more complex pipelines, more, more better models, better data. And that's actually what they tried to do for the Ulta Feedback paper, where they initially sourced instructions. So a lot of both synthetic and human-generated instructions from an instruction pool and actually asked, prompted different models to actually provide completions for each one of these instructions. And each one of these completions was actually judged by GPT-4 based on 4 different criteria. So instruction following, truthfulness, honesty, and helpfulness. And also one overall criteria, just asking or prompting the model whether overall the completion was correct according to the initial inputted prompt. And as you— a good thing to take away is that whenever you use do this judging, what we've seen is that each of these individual criteria, when you're focusing on instruction following, truthfulness, etc., highly correlates on average, so to say, with one overall rating. So if you actually want to reduce some cost and reduce some compute, you can also just apply one overall rating. Also, given the fact that you probably need some, some human involvement at the end. So also this didn't turn out to work. So when we actually started uploading this data in our UI and actually looking at the data by, by human judgment, we actually saw that some data was sourced from benchmarks, that there was like this weird thing happening where there were some prompts that completely— some responses that completely didn't make sense and still ended up getting a very high score. And we kind of investigated that and investigated that and looked into the code. And it was apparently a coding error leading for the scores to be converted from a 1 to a 10. So all of these scores that were actually a 10 could have been, been a 1. So that kind of messed up the entire dataset. There was like incomplete ratings due to the fact that OpenAI API calls failed and there were some ties in the data that were still being processed as chosen and rejected pairs, even though if the same response— if the responses would get the same rating, you wouldn't expect them to have one preference over another because they would be in a tie. And that's actually what we kind of also showed, that whenever you spend some time, look at your data, get involved, and work towards your higher quality dataset, that you can actually train a better model. And initially the Hugging Face team published the alignment handbook, saying that this awesome Zephyr model and made the entire research reproducible. And then we did that exact reproduction, but then instead we used the cleaned Ultra feedback dataset and that actually led to a model that performed better on the benchmarks and also intuitively, according to human judgment, performed better. So then apparently you need, yeah, better data or better synthetic data. But whenever you actually want to work towards that, you— there's a lot of things that, that come to mind. So initially you can kind of start spamming ChatGPT with random requests, but if you actually want to scale that or make it more cost-efficient, want to avoid vendor lock-in licenses that you might not be able to be allowed to use all of the different models for actually generating synthetic data to fine-tune models, some fault tolerance that might arise, as you have seen in ultra-feedback dataset and things like, like structural generation where you might want to just output JSON output responses in JSON format. So overall, there's just a lot of complexity involved in actually starting to do this. And yeah, I've chosen a set of tools that you might kind of look into whenever you start working with your own synthetic data. And one of them is the Outlines package for structured text generation. And this can actually produce like structured text generation based on like prompts for LLMs based on regex, JSON, Pydantic models that everyone loves and uses, and also actually functions to be able to do function calling. And this actually relies on the modified— by modifying the sampling of tokens. And yeah, this is a very accurate approach to actually take. And it actually also reduces the inference time due to the fact that there's a limited number of tokens that you can actually sample from. And one of the interesting blog posts from Hugging Face actually dives a bit deeper into this topic. Another package is DSPy that probably everyone is familiar with, but it's focused on programming, not prompting, so to say. And this actually lets you define a DSPY signature that you use for actually calling an LLM where you kind of program the expectations that you might have for your model. And then whenever you kind of optimize this for your specific use case, the package underneath tries to optimize the prompt and optimize or fine-tune the model weights slightly so that then eventually you end up with a higher or more accurate prediction. And one thing that Hamel Husain actually showed in one of his blog posts is that, yeah, under the hood, the model or the DSPY package actually makes hundreds of calls to the OpenAI API when it tries to optimize these prompts due to the fact that it's gathering good few-shot examples for your use case. And that actually is quite costly. And that's an interesting blog post where Hamel kind of goes over some of these tools as well. So I recommend reading that too. And the last one is DizzyLabel, and it's a synthetic framework for synthetic data generation and AI feedback. And it relies on, like, a directed acyclic graph structure and is more of a pipelining framework where we kind of worked on serializable pipelines, cached intermediate results, and actually included structured generation as well based on the outlines package, and everything is executable in parallel. So whenever you really want to scale this up, then you can actually do that. And it's kind of like application development, like the things like LlamaIndex or DeepSAT or some of these tools, but then really focused on the dataset engineer. And one of the interesting things that I've seen come by recently was also this blog post about the rise of the dataset The Data Engineer. So not really having AI engineers or these kind of people anymore, but really people that focus on data and data quality. And it's an interesting read as well. So when it— for your working on improving your data, you— it's like an iterative process where you have your original dataset and you add some things to assess diversity, the dataset quantity, dataset quality. You do deep duplication filtering and these kind of things, and you throw some like human feedback, but also AI feedback in the mix to make it like more and more intuitive and more smart. So what you can actually ensure, or what you can think of for improving your dataset quality, is that there is enough quality, but enough quantity, but more isn't always better. So it's also fine to throw away data that's maybe very present within your dataset that you might be able to deduplicate or also reduce to— simply reduce to lesser quality examples within your dataset. So one example, what— some examples that we've gathered from actual public reproductions of these initial fine-tuning techniques that Daniel highlighted earlier is that for SFT, we've seen great results according to benchmarks with 10K input samples for ORPO, 7K, TPO, 3K, and SPIN as low as 2K. Data examples. So apparently with a lot less data and a lot of models are being fine-tuned, you can actually do a pretty good job at achieving high benchmark standards, but probably also to actually have some good human intuitive models. And whenever you do think about data duplication, there's a lot of research being done in duplication, but often the duplication pipelines aren't very— deduplication pipelines aren't very reproducible and not very well documented. But yeah, as we've seen for the FinRep dataset that's recently been published by Hugging Face, they actually published the entire pipeline and made it publicly available to their Data 12 library. And what you can think about when actually doing deduplication is applying intuitive rules, like applying existing metadata that you might, might use for filtering your data, for filtering out some irrelevant examples for, I don't know, doing something like topic-wise deduplication where you might want the most representative examples per topic. You can also think about creating your own custom metadata or your own custom features and doing intuitive similar things like hashing of the strings or doing filtering based on embedding similarity and then actually grabbing the exemplars from the embedded data points that are most representative for your data. When you're doing rule-based data, it can be, yes, rule-based cleaning of your data. It can be as simple as just writing a regex query. So for example, querying as a large language model, if you don't want that to have that in your dataset, normally some models used to respond like this or using the word Delve or using like these quotation marks that that might indicate that the model is going to kind of hallucinate random references, some random URLs that you might want to at least change within your dataset. And all of these things are like very intuitive, human-understandable rules that you can apply to in order to improve your dataset quality. And on top of that, what you can also do is, yeah, apply these more advanced techniques or more advanced classifications or embeddings or these kind of things where you quite easily can get a zero-shot model out of Hugging Face that's performing quite well and decide some topics that you deem relevant for your dataset and actually start predicting, making initial predictions for these topics and actually use that as some initial ways to filter your data as well. And these classifiers can also be very, very cheap. So it's— you can actually use one of the zero-shot models. You can use a set-fit model after annotating only a couple of examples. And if you actually want to go more expensive, you can look at LLMs as judges, juries, or rationalizers, where you also have this rationale, like the explanation as to why this— they provide this score. And if you want to go a bit more simple and a bit more intuitive and explainable, then And something like text descriptives, like the package where I provided the URL, can provide a very easy out-of-the-box analysis of your data. And all of these methods that I went over are actually intended to be used, according to our vision, our view, along with human annotation. And this really helps normally to make human annotation more fun, more engaging, and also more, more effective. So that's kind of where you want to be. So maybe somewhere in the middle where you can either choose for a very basic, simple, intuitive approach using Google Sheets, or like a completely customized approach that really works for you. But for, I would say, for, for a lot of the people, something in between where you might use like some already out-of-the-box tools or web apps really works and really differs per person what you want and what you prefer. So one of the custom tools that I've seen combined to kind of play around with your data and kind of see what's in your data is this bulk annotation from Vincent Warmerdam. And it's like a custom tool that he built with Bokeh and Pandas and some JavaScript to tie everything together where you embed all of your text and you can kind of explore and kind of play around with it. So this is what it looks like. You can do like this lasso movement, kind of see what's within that specific cluster, and then move from there to kind of annotating and actually saving your initial results.

By the way, I have a question about this. I saw this too and I thought it was pretty cool. Does Argila have something like this? Specific functionality.

Yeah, we, we actually have something like this where you might be able to attach vectors to your records and you can do semantic search, but we don't have this interactive kind of annotation thing. It's a thing that we've considered. I believe Snorkel has something similar to it, but we've not gotten around to actually implementing this. An alternative that you might consider is like using something like notebooks where use notebooks to initially annotate your data. And I think everything in terms of notebook annotation was a request from Hamel Husain to kind of highlight this is based on IPy annotations. And then from that, there's a lot of branches kind of looking roughly the same and giving the same overview. But this is also fully customizable where after you've filled out one annotation, you can actually do some callbacks and actually do some post-processing. So we might be able to do some active learning and actually start gathering data for fine-tuning classifier on the fly, so to say, and then run inference with that classification model. Another thing that's kind of like a little bit more out of the box is like creating web apps with, with Gradio, with Streamlit, with R Shiny, and you can actually use like all of their components and tools to directly out of the box have a very intuitive basic UI. And this can also be fully customized normally. So there's one UI of the, the Summer NLP Hackathon that someone created that we normally host together with Hugging Face. And this is the input text where after that someone is able to kind of correct or say, okay, this is the right translation or wrong translation. It can also be used for like the KTO example that Daniel mentioned earlier where you might give a thumbs up, thumbs down and kind of go through your records one by one. And this is once again like a nice, nice example, but with Gradio, Streamlit, and Shiny, you can really customize everything as far as you might want to do that. And if you actually want something a bit more out of the box, there's also, you have these fully fledged cool solutions with a lot of integrated features. One of them is Lilac, and Lilac kind of has this dataset overview thing where whenever you select the dataset, you can actually select different topics and you can actually see some of the main clusters within your data, and it's pretty cool. And on top of that, they also have this semantic search feature that I previously mentioned. And on top of that, I'm curious about Lilac, by the way.

Do you recommend using Argeela with Lilac for like the different features, using them concurrently, or like what's like the stack that you like personally?

For me, it's kind of the A bit, a bit of a mix of both. I think it really depends on what you prefer, what you like. I find the Lilac UI a bit less intuitive personally. For example, for within, within Argila, there's less, I think, features included and the UX/UI is a bit more intuitive for me. Things like, like spacey explosion I played around with. But it really depends on what you want. Probably if you as an engineer look at the API that we have or the SDK that we have compared to the one with Lilac, if you as a domain expert or labeler kind of go into, into Lilac and find their features and UI UX better, then, then go for that. But I think it's the same tool inherently covering the same topic, having a lot of the same or similar features. And for me, it really depends on, on that kind of thing.

What about you, Daniel? Do you have a, you have an opinion or do you have a stack that you like?

Yeah, I mean, I think one thing is like at what point in your kind of process you're trying to look at your data. So I think there's this like initial kind of vibes check where you're trying to look at quite a lot of data very quickly and maybe looking at a few examples carefully, but not necessarily like actually fixing things yet. So I think it's good to kind of separate the kind of annotation mindset from the, like, I'm trying to just understand this data overall. And I think for that, Lilac can be quite nice because you have these different kind of visualizations. And I think the other thing that's, like, super crude, but if you can get your data into some kind of data frame, even in Google Colab, and compute some features like string length and token counts and very, like, basic crude features, it gives you these plot suggestions, which are actually, like, sometimes sometimes okay, and it's just like a really lazy way of being like, hey, there's like this one really random outlier. Um, so it's a little bit crude, but I think it can be quite a quick way of finding like where something's going wrong. And then I think after that, kind of trying to dive in a little bit more detail and actually poke around the individual responses.

Yeah, and I think with the, the, the nice thing from, from Lilac or the more full-fledged tools is that, yeah, you don't need to think about what you want, so to say. So of course they have a lot of these highly opinionated features built in, and based on that, you— I think a lot of people will get away with having like some nice out-of-the-box toolkits. Uh, there's of course always people that are going to want to have some, some custom thing built on top of some, some custom features, and then I think it's— if you really value that, and it's also worth spending time on that and actually building some of these custom tools or some of these custom annotation flows out of the box. I think both Lilac and Argella are open source as well. Yeah. So this was the other example that I had. Argella, same kind of idea, having a dataset overview. You can log in and play with a demo as with Lilac. And also you have this semantic search kind of thing. You have your like content that you might load into your UI, and you can actually start annotating and doing bulk annotation and all of these, these kind of things. So it's same, same but different. Don't— I have a preference with for Argila, of course, and I'm more familiar with that, but I guess for everyone it's up to them to decide which, which UI and kind of API and workflows they prefer. Yeah, so maybe I'll hand it over to Daniel again. Stop sharing.

Okay, can you see that? Yeah, perfect. So I'm just going to talk through some example datasets, and this might seem a little bit weird, like why I'm talking about these very particular datasets, But I actually think you can sometimes learn very little, but sometimes learn a lot from looking at some examples of datasets people in the community have built, particularly for like approaches to generating these datasets in kind of creative ways that doesn't just rely on either throwing loads of API credits at the problem or loads of human effort. So I think that also goes back to this this thing I was saying earlier about DPO datasets, why they became so kind of popular in the community is because this idea of generating a chosen and rejected pair can be done in like a lot of creative ways. So there's an example here of this dataset that's trying to make models, large language models, better at writing. And the approach they take is basically to kind of take some existing books, written by humans and then prompt a large language model to summarize those and then prompt another model to rewrite based on that summary. So, you know, write a novel or chapter of a novel based on this kind of summary. And then they choose the kind of the original human-generated response as the chosen and the model-generated responses are rejected. So that might not apply to your use case, but I think it's an example of like how you can get at these things without having to, to just rely on either human data or, or model data. Here's another example. So this one is actually kind of a vision model, but it has a kind of similar interest in that it's focusing on kind of actually generating this data flywheel, so trying to get this thumbs up, thumbs down. So I mean, it really depends what kind of context you're working in, but I think getting these thumbs up, thumbs down ratings can be quite useful. And the other thing that I found from doing a lot of annotation, so I think it's worth thinking a little bit about the kind of ergonomics, so not just of the tool, but also what is the task that you're actually trying to do when you're annotating and how does that work better. So for these kind of preference datasets, depending on what you're trying to do, sometimes it can be really easy to say that's a good response or that's a bad response. Other times it's really helpful to have like these two generations next to each other and say, okay, this one is better than this one because like without having a comparison, person, it's quite difficult to actually say which model response is better. So I think when you're doing this human annotation, I would give like quite a lot of thought to those kind of questions, which I think people kind of don't really think about that much, but I think it can actually, yeah, I think it can be quite important for both your experience of doing the annotation and how pleasant and easy you find it. But I also have no evidence for this, but I suspect it actually influences the results of the annotations you get quite a bit depending on how you out the task. So I think it's worth giving a bit of thought. And I think what I maybe will do is just give like a very high-level overview of this kind of project I'm actually working on for this course. So that is basically trying to build a large language model summarizer and hopefully a small large language model summarizer so it can actually be deployed easily. That will take a dataset card for a dataset hosted on the hub and turn this kind kind of long markdown into a too long, didn't read summary that you can kind of very quickly look at and be like, what is this dataset about? What is it for? This is just like a very kind of high-level overview of the steps that are kind of taken to kind of create that dataset. So you have, as I kind of mentioned at the start, a bunch of processing that you do with the markdown, and that will also be processing that you'll carry over to the actual deployment. In the markdown, you have a lot of stuff that you probably don't want to even pass to a large language model because it's just like formatting or content. It's not very informative. You might want to move some of this YAML. And then you kind of get this summary. And then in this particular case, like the approach I'm trying to take is to build this preference dataset. And there's different ways of approaching this, but this is what I'm kind of using this Distill Label pipeline for. And this kind of looks a little bit crazy, but I guess the overall workflow, which is actually not that difficult to define, is that you load this dataset that has a bunch of these dataset cards and you do this kind of initial filtering, you format this prompt. And then in this case, I'm trying to use OpenModel, so kind of compare these 3 different summaries. And then use ultra-feedback, which is a kind of approach to doing this language model judging. And in this case, I'm using Llama 3 for that. And I guess the kind of thing I want to say about this pipeline, I think with synthetic data generation, it really depends on what you're trying to do. Sometimes just like having a prompt and a bunch of inputs and being like, right, just call that API a bunch of times with this prompt will work well. And sometimes this kind of more complicated pipeline can work better. But the thing I found quite nice about this is that each of the kind of steps, you kind of maybe increase the number of responses you're generating. So maybe initially when you're comparing these different models, you kind of do like 100 or something and just kind of very quickly start looking through the responses. And I guess at that point, what you're thinking about, which is kind of similar to this point that you often do when you're kind of thinking about using prompting versus fine-tuning is being questioning about, okay, can I just improve this via the prompt? So you might have already done a little bit playing around, but when you kind of get a few more responses, you think, is there something that I'm getting in the response that can just be fixed in this prompt step? And you kind of iterate on this pipeline, but each of these steps might actually remain, like, the kind of workflow might remain quite static, but you kind fiddling around with each of these steps. And the other thing that I will say in this kind of particular pipeline, I'm sending all of these responses to Argila. So you're kind of looking at two things when you're doing that. So you're looking at the model responses and seeing how they look. But the other thing that I'm trying to look at when I'm kind of building this pipeline is this LM judge part, because I'm basically assuming with this workflow that this judge is actually going to be able to detect which summaries I'm going to like better. So one of the things that I kind of do, and there's a kind of notebook in GitHub that I'll share afterwards, but basically write a bunch of examples and then look at the ratings that the model gives and whether there's some like average better rating, but also like how often do I change the rating from the model and like how big is that difference. Maybe if it's like only one score different, then that's not so bad. But if it's consistently getting bad ratings, then that's maybe something to worry about in terms of whether it's reflecting what I want the LLM to do. And the other thing I would say about this part, and I'm probably a little bit overly obsessed with this idea, but I think when you're looking at these ratings and the responses, like thinking about heuristics that you can use to kind of capture, is this response good or bad? Like maybe it's too complicated to judge via a simple rule, but quite often there might be patterns that you start to see. So once you have like even 100 ratings or maybe even like 20, you can start being like, okay, do I always prefer the one that's shorter or the one that's longer, or is it like random? Because if it's just always a shorter one, then maybe you can already say, I just need to like cap the kind of maximum tokens or do something a little bit more manual. Or I can just basically say all the short responses are always better. And then skip this kind of LM judge part because this is the kind of expensive part, but it's also like slightly brittle because you, in this case, like put quite a lot of faith as you kind of scale this in this consistently working well. And if you can use some of these simple rules either instead of or as well as the kind of judge to check, okay, is this continuing to work? Well, I think that could be quite useful. So because we're running out of time, I thought I'd just quickly talk through this repo that we created. So there's some notebooks in here that kind of give some examples of like different approaches to deduplication. And how to kind of do these, these kind of database checks. And then an example of using Distill Label to generate one of these synthetic pipelines. And the other thing, which I have to admit is a work in progress, is try to basically put all the work I'm doing for this particular project in here. I'm not saying it's necessarily like a good example of what to do, but I will at least try and share it openly and kind of communicate what I'm trying to do there. And then, yeah, I'll maybe just quickly point to these other resources that we have here. So one thing I just wanted to mention is that with the course credits that you have from Hugging Face, you can use that for Spaces, but you can also use it for inference endpoints. So I think between all the course credits, you could actually generate quite a lot of synthetic data. And I think they're quite interesting to see. Kind of what people can come up with that. And then, yeah, beyond the kind of notebooks for this session, there's also a few other resources here that might be interesting, including this blog post about FindWeb, which talks a little bit about their approach to deduplication. And it is kind of for a slightly different use case, but I think there's some kind of nice lessons in there that might still be kind of relevant that, you know, doing like a really aggressive deduplication doesn't always work better. And there's these kind of subtle heuristics, which unfortunately, like like often you have to actually train a model to see what the impact is. But if someone else has done that, that is quite useful to see if there's something you can pick up there. So yeah, I think maybe—

sorry, go ahead.

Sorry. No, sorry, I was just rambling to a close.

No worries. We have about 2 minutes left. There isn't too much time for questions, but I'll just throw one out there. Aminov asked, how would you recommend generating synthetic data if we're fine-tuning on proprietary datasets and human annotation is expensive? Proprietary datasets that someone else creates, like a vendor of some sort, or Yeah, I mean, I don't really know what he means, but maybe it's, yeah, like fine-tuning on— I don't even know if the word proprietary really is. I mean, just like your own company's data, just like, yeah, let's put it that way.

So I, I guess it kind of, it's kind of the same as whenever you would use or fine-tune your own model versus, uh, fine-tuning or versus using a proprietary model. Um, it's, yeah, about data ownership by privacy, about the fact if you value these kind of things in your company, if you need these things in your company, and if you really want to be the owner of your data and your model. And those are, I think, the main takeaways for this kind of trade-off, both for models but also for your data.

Sounds good. We can probably take the rest of the questions to the Discord. Maybe y'all, if you want, if you, David and Daniel, if you have time, you might want to peruse it. There's a channel dedicated specifically to this talk where people are in. But yeah, thanks a lot.

This is a very good talk. And just to say also, like, I'm like overly excited about people building datasets. So like, I'm happy to try and help out. Like, I might not have the answered, but I'm happy to like brainstorm with people if they're interested in building a dataset as part of this and sharing it, then definitely try and give them a hand with that.

Yeah. Great.

All right, thanks a lot.

Thanks. All right, thanks, Fabio.

Bye.