Getting the Most Out of Your LLM Experiments
Thomas from Weights & Biases shows how to track and debug LLM fine-tuning experiments using W&B, Axolotl, and Hugging Face. He demonstrates dashboards for comparing runs, logs batch-level data to catch loss spikes, and introduces Weave, a tool that traces LLM API calls and logs execution for evaluation. A live example uses Weave to solve the New York Times Connections puzzle.
Chapters
So hello everyone, thank you for having me here. I'm Thomas, I'm a machine learning engineer at Weights Biases. I'm based in France. So let me tell you a little bit about myself and what we're going to talk today. We're going to talk about reproducibility and how to get the most out of your experimentation. I am a machine learning at the growth part of Weights Biases. I'm mostly working on fine-tuning. I became kind of the fine-tuning guy at WandB. I'm working internally with LLMs and externally trying to help customers get the most out of the fine-tunes. And all the problematics you have been showing during the course, actually, they're very real on a customer basis. I also contribute a lot on the open source ecosystem. I've built some integrations. I contribute to the open part of Weights Biases, like the examples repo and the Axolotl integration. Some of the Hugging Face Transformers bots. I recently collaborated with the TorchTune to have a Weights Biases integration in there. And yeah, I also like work with our new product I will show at the end that has Mistral integration already built in. Previously, I was working more in the traditional ML part. I was working with time series data. So maybe some of you would relate with geospatial data, some computer vision with satellite imagery to actually create models to forecast renewable energy production. So some electrical engineering with the data science, data analysis, and then like deep learning actually. I'm also a fast.ai student from multiple versions. I collaborated back in the days in fast.ai 1 and 2, and I really like this open source and Discord community that has been formed. So yeah, I want to just start by saying that keeping track of everything you do on this field is complicated. In computer science, it's generally complicated to take care of everything, but here we have hyperparameters, datasets, now we have prompts and we have to run evals. And like, I feel like this guy multiple times, like I need to bring my Docker image, I forget how to run that, put on a compute and maybe a switch provider. So like, you need to be like a Swiss knife here and learn a lot of different toolings to be able to like to port your code to different environments. So we at Weights Biases are a company that started like creating tools for experiment tracking. And as you have seen during the course, there are multiple users. Hamel Husain showcased the Axol integration. Wing also came and showed how to use it. I'm not gonna Most of you know what Weights Biases is, so I'm going to just point out that we have Axolotl integration, and most of the fine-tunes these days also happen on the Hugging Face trainer. So we also have a very simple integration there. It means that you just have to pass a flag and you will get your dashboards, these beautiful dashboards with the metrics logged to the central repository of metrics. So instead of like just talking about that, I'm going to show some of the work you have been doing. As someone pointed on the Discord, I have been trying to help people around and how do I do that? Because people share the fine-tuning experiments and then I can like sneak peek and try to help them out the metrics, what they are logging, maybe something is failing. So we have this reproducibility baked in when you use a product like this. So just to show, shout out some of the users here, that one show is trying to like his first fine-tune and using like Alpaca dataset and trying to run Llama on top of that. Yeah, Hamel Husain set up like a fine-tune channel, sub-channel that you have like threads and people can dig in and share their experience on their fine-tuning journey. So I've been like sneaking in there and trying to see what people are doing. So we can actually click and see what Damon is doing here. And yeah, he's actually performing some different fine-tunings here, maybe tweaking some parameters. And as you can see, he has run multiple experiments. And he has like highlighted 3 of those for us. Like this little eye lets you toggle on and off different experiments. As you toggle, the runs like pop in and pop out. It's very dynamic. And we can actually dive into one of the experiments and maybe the metrics are organized by train and eval here, the learning rate, the loss is kind of going down there. And we can check on the overview tab and see like, oh, he's using Axolotl. So this was run with with the actual CLI train, and there is like a YAML file with the config, and we actually grab the config automatically for you. So here you are, there are the parameters he used for this fine-tune.
I think config is already also in the files now too.
Yeah, yeah, it's actually a lot here. Like there's an artifact here where we have the actual raw YAML, so you can pull that and run the exact same experiments he's doing. So this is the actual config, not bars, like raw YAML. I think that's really useful because you want to actually pull the YAML and then like rerun that experiment and maybe tweak something. So having the actual YAML as a first-class citizen here is very useful. So yeah, he has run multiple experiments. That's really cool. So we can actually see what he's doing, how the runs are performing. So to come back to the slides, yeah, another cool project is Zach. He was trying to do some self-instruct here and he was like sharing some knowledge on this like grokking points kind of at the end of the epoch. We say that maybe memorize some of the data and that's why we have like these dips on the loss. He was trying to improve his experimentation, tweaking some parameters here and there. There is a link to the actual dashboard if you want to check it. I feel this is really interactive.
Or what was interesting about this experiment?
I think it was, he was actually tweaking some stuff on the trainer himself. I don't know if Zach can tell us more about this experimentation, but he has multiple projects here. One of the ones I linked here is this one, but I think the screenshots are not from this project exactly. But yeah, we can see that actually he's running on with a lot of like override on the fly with, he's probably using the trainer here, straight trainer. And he's like running on an 8H100. You can see that, yeah, it actually trains pretty fast. So it's, yeah, we can, yeah, he was actually toggling off. I think some of these runs crashed. So if you go to the runs view and see on table, yeah, some of them fail or crash. Probably he could kill them before finishing. Yeah, the long runs are like 6 hours here. Check one of those, like this one. Yeah, he was trying to like tweak the parameters to actually see if he could avoid this small, like, as you can see, this one has like way less, but I'm curious if he like actually, no, the learning rate's still pretty reasonable. Like, because one technique you could do is like put a super low learning rate, but that means your model like learns slower. So it's not always like the silver bullet here. So yeah, that's, Some of the experiments like Zach tried, there's a big spread there.
Are these plateaus, are there, is this related to the, like the Answer AI post about?
Yeah, this relates to the Answer AI. Yeah. Yono and Jeremy like did some experimentation about like trying to memorize one batch and actually seeing this, but this is like a longer QLoRA experiment he was doing. Yeah. She was trying to actually, I don't exactly know what he was doing with the prompt here. But we can ask him on the thread. I encourage people to point. Other cool experiments we have seen is that, yeah, someone is trying to actually teach kind of a better function calling to Mistral. This project is not public, so probably there's like a curated handmade dataset that people don't want to share. So to actually be able to share the configurations he was using, he created like a something we call a report in Weights Biases is that he's showcasing only a part, a snippet of what he wants to show here. So we have like the actual config. I think he was running into issues with the Axolotl config. So he wants some debugging. So he pasted the config here. So on some of the training metrics, I'm curious about this because there has been like other models, like, you know, Technium train, like Hermes Pro has some function calling functionalities before, Mistral added function calling to the instruction models. And there's like a lot of forms you could code the dataset to actually support function calling, maybe adding special tokens or not. So yeah, this is a project to take, keep an eye on it. And maybe if they push the final checkpoint, we could actually test it and benchmark against the other models that have function calling. I don't know what do you say, Hamel Husain, but like, yeah, Now this is, he's not running the Mistral 8 7B, as you can see here, he's running from the Mistral 7B, but I think this was like some experiment naming that he put at some point. I'm curious how good a 7B could be at function calling without losing too much like knowledge on the previous data you showed him. Like you need to show a lot, a lot of function calls to be able to actually learn that. So. And yeah, what other products we have? This one was pretty cool because he actually shared like a project we built internally at Weights Biases. This is a user that has been like training a lot of Japanese and yeah, trying to get better bilingual models in Japanese. I have, I have worked on Japanese data previously with GPT-4 and it's good, but it's not always perfect. It's way worse than, than other languages, like Latin languages are way better. Sometimes it's like it messes, like when you have like 3 subjects in a sentence, it kind of makes them up. So yeah, there's have been efforts like Naver in Korea and this to actually have a specific language model for Japanese, Korean. So this is really cool. So he show, he link, sorry, he linked like the leatherboard we are trying to build with the Korean team. Korean-Japanese team to actually get a good scoring of the capabilities for those models on those languages. So you can check it out. He's a very seasoned fine-tuner, so he has a lot of experience. And yeah, of course, during the course, multiple persons have showed like Weights Biases dashboards or talk about like logging. Yeah, Jono is like a user of Weights Biases for a while. Sophia showed that they built an integration on the fine-tuning API to actually enable you to log your, your experiments there. Hamel Husain showed some integrations with Axolotl and how you can log your metrics there. And other people outside of this course, like Karpathy, uses, use it on NanogPT. Mark is using it. Wing use it a lot to debugging all the Axolotl improvement he's doing. Jeremy has shown some metrics. Probably it was not Jeremy that built that dashboard, but I know Giorno and Benjamin use this internally. And yeah, Maziar, that he's in the course, he's also like pushing a lot of checkpoints and good models to the Hugging Face Hub, and he's also using Weights Biases to keep track of the experimentation. So I want to show you like another dashboard, something I built recently that it's like a bigger project that maybe you could relate to that. This is like an experimentation we did with with a couple of persons actually from Discord. We create an open source group. Jono was there at the beginning, then he like helped it out at the beginning to actually organize this experimentation. And we were doing some ablation studies on Mistral 7B. So we were removing layers and retraining them all on two phases like SFT and DPO to get them all back to talk because once you remove layers, because we were removing like 75% of the layers, the model was broken. So in this project, you see like there is a way more like complex dashboard in the sense that, yeah, we're grouping the runs. That's something you could use. Like if you have runs that relate to something, maybe grouping is useful. So in this case, we have the same initial model, like one stage, two stage, and then the evals. Another cool thing you could do here, it's actually having different views of the project. So this is like the general dashboard that is very kind kind of meaningless because it's very bloated. So I have like a special view that organizes the dashboard in a way to just check the first stage of the fine-tuning. Go back to the last screen.
One second. Sorry. What is the— go back to the previous screen you're showing.
Oh, that one?
Yeah. What is this like shaded area? What is that?
Yeah, because when you group, like it takes the it shades to get the full curves. It takes the average of the curves. So it doesn't make much sense here. So this is like kind of a dashboard that is not very useful because it's trying to compare like the train loss of two different training recipes that you cannot compare actually on the same dashboard. That's why in this project, like there's like the default view, but it's not very meaningful. Like I have an SFT view that only filters out like we're using like a regex here and filtering out only the SFT runs, and then I can compare the training losses together.
I think of, um, so I think of Hugging Face tools as being really nice for these ablation studies. I don't know whether you get to see very closely what people are doing when they're doing these ablation studies, but if so, is there anything that you've seen that jumps out? And now when you talk to new customers, you're like, oh, I saw this pattern and it would be like a really cool thing for you to do in the future? Or is there like any workflows that you think, or insights for people who are doing ablation studies that you think other people should know?
Yeah, this project has a lot of context. Like I can actually put the link on the Discord. It has like a prezo of 1 hour. It's actually, at the time we built this, this was when Tiny Llama started training like a small Llama before Phi, before before this small model. So the idea, the initial idea of this was let's create a really small model from the big model, not training from scratch. So there is like papers on that, like ShearLlama that actually, yeah, do like smart ways of pruning the model. This is like basic, just removing layers completely. We actually, Jeremy at the beginning suggested some of these strategies and And yeah, he said like, just remove the full layers. Like it doesn't matter if it's like not that deep and it's shallow. So we started playing with that and then we kind of abandoned the project and then we came back later. We got some compute and we were like, okay, let's see if we can get a small model because then drafters came out. Like, so we could use a small model that it's as similar as possible as the big one as a drafter. So we were like, okay, let's try to make a 2B or 1.5B. For the Mistral Instruct, but we don't have the recipe for Mistral Instruct. So here we use the Sefi recipe that at the time was one of the best instruction tuning recipes. So that's why it has two stages, like SFT and DPO. This is actually applying the same Sefi recipe. So you destroy the model and instead of doing continuous pre-training, you recover the model just straight with instruction tuning. And it works. The final model has a reasonable score. It's of course not as great as the big model, but we put like a super limited budget of 1 billion tokens. That's very, very low. And we were able to have a model that drafted the big model and they kind of work. And a fun fact, like during the Mistral Hackathon, another team was trying to build like an embedding model from the Mistral 7B. And I talked to the team. I was like, oh, but I did this. You can try the Prune model. Maybe it's good and better. And they were trying to do something really fast. And actually the model is very good and better. So yeah, there's multiple usage of things like this. Of course, it was like a research project, kind of not targeting like deploying this model, but yeah, it was good insight. Like we were not expecting actually to recover the model so fast with instruction tuning. And yeah, also we try a lot of stuff automatically here. So like when you push the updated, like we were, we We create a script that ablated layers, and then you push the checkpoint to Weights Biases, and we trigger automatically all the steps like the supervised fine-tuning, the DPO phase, and then we trigger eval harness. That's like the framework from EleutherAI that runs the evaluations. So I also have like an evals dashboard here that is like organized for evals. So we have like the standard eval harness metrics that will come back to the dashboard. So as you can see, yeah, the big models are way better and like the small models are slightly worse. No, half as good, but reasonably good for a model that couldn't speak English before training. So yeah, that was like what I want to show you here. This is like a more complex workspace that has all the bells and whistles from Weights Biases that I will not dive here, but we can discuss this later if you someone pings me. And yeah, that's what I want to show you. And maybe like, yeah, another trick I want to show you here is that some people like in a dashboard like this one, some things that people don't know that they can do is that sometimes you have trouble like inspecting your runs here. And instead of showing you a project of mine that has it, I can show you in this project that's from Dan. Yeah, you can see this runs view. And most of the time when I'm interested like in a metric here, I will actually ping some of these metrics. Maybe, I don't know, I want to see this parameter and you have the spin column option that will let you like bring the column to the left. So you can organize your workspace and that metric is going to be always visible. And another cool tool I may, you may not know, I'm not going to show you everything, but there's like, I also find that for for you guys doing fine-tuning experiments on this course, the run compare is very useful. That actually, yeah, shows you the difference hyperparameters for all the experiments you have done. And actually you can toggle diff only. So it's only showing you the difference between the runs you have toggled on and off. So if I toggle that on, it will like re-render with that. So in a bigger project, Maybe this one, like I have a big project I did with Morgan a couple months ago with fine-tuning Mixtral when Mixtral got released. So we were using this run compare to be sure that we were not messing up and the version of Axolotl was changing. The transformer underlying version was changing. We were like pushing fixes here and there because like there was not an actual Mixtral fine-tuning recipe there established. It was too early. So we were looking at the difference using this run compare table. And like, so that may be useful if you're doing a lot of experiments.
How do you activate this comparison table? I haven't seen that one before. Is it?
Oh, it's just, if you are in any project like this, you add a panel here and it's like run compare and—
Oh, okay. Oh, I see.
And then like, I always toggle diff only. So I don't want to see like the things that are similar because yeah, LLM training configs sometimes have like 200 parameters. So I just want to see like what's difference between the training runs.
I have a few questions. Some of them are from the Q&A. I'm going to start with one of my own, but then I'm going to try and get through some of the ones that are in the Q&A. One of the things that, yeah, a lot of these graphs are very focused on showing loss, the eval loss as a function of something else. And especially in the context of conventional machine learning, like we always just like loss is a quite good proxy for how good the model is. With LLM fine-tuning, I've never been really confident that loss is a proxy for output quality. And as a result, I spent a lot of time inspecting the output of the model. And I haven't formally, like, looked at the correlation between having a lower loss and me subjectively think the output is better. But I have, like, an intuition that it's not that tight. Do you hear about this? And do you see that people are frequently trusting the loss is a really a great measure of model quality, or do people, do you see a lot of people are just inspecting, visually inspecting the output samples?
I think you have, during the course, you have covered a lot of the evals, and I think it's really important to build an eval dataset. And with generative AI, like with these LLMs, you need to generate samples. And Ideally create kind of a test that enables you to compare if the model is better or worse and kind of what Eval Harness does. It computes metrics on different benchmarks, but that's like for a more generic fine-tuning. In your own use case, you will need to compute your own benchmark for your own task. But yeah, you need to look at more samples. You need to put them all in inference and run those benchmarks. Actually, when you run this benchmarks here, this project, like, yeah, there's all these hidden sections is because we have the actual samples there. So you can go and see what the model answered on the MMLU, whatever, example number 5, and see what the answer was. So you should, you could use Eval Harness, create your own task and put it in the format of Eval Harness and it will work because we have an integration with Eval Harness. But yeah, I don't know, like we have developed some LLMs, not fine-tuning, but internally. And yeah, basically we ask ourselves to create our own data, like 5 question answering by engineer, like times 10, you have 50 twice a week and you curate easily a dataset of a couple of hundreds of samples that are high quality. Yeah, score your model against that. And during training, yeah, I suppose the loss is a good proxy of things going wrong, but it's not a good proxy of like, if you have like 5 fine tunings, you tweak 3 parameters, you will get basically the same loss. And maybe one of those is better. Like it's not easy to know. I don't know what's your experience.
So, uh, you're asking what Hamel's experience— my experience is I just look at the samples.
I, I think that, yeah, like I said, um, I think it's powerful looking at this, at the samples, and actually vibe checking that it's correct. That's a good proxy of the things getting, getting wrong. And actually, you need to look at the samples because sometimes there's like formatting issues that you are not aware, and like you are— I don't know, there's so much abstraction about like how the token, you actually need to look at the token ID sometimes, decode the batch and see if, I had a problem a couple weeks ago of a model that was spiking at some point and to actually see what, what was going on, I decoded the batch where the model spiked and there was like some characters are not on very out of the distribution of the tokenized. So there was some Korean on an English-only dataset and the model just spiked there every time, even if I tweak the learning rate or whatever. So go back in time, rewind, decode that batch, see what you are feeding the model. It's easier said than done, because like if you have 8 GPUs and you have distributed sampling, so at some moment, like keep a checkpoint, rewind in time. And, but this library is like Transformers as well, lets you do that. Like lets you like index on the DataLoader and see how you are feeding the data to the model. Yeah.
Data in this interface because it's too restrictive for me. So I end up building my own data viewing tools. But yeah, I mean, that's what I do. But yeah, I agree. It's, you have to look at these data points somehow.
Yeah. Yeah. Yeah. And you like, I think you got, you have to look at both, like even the token ID sometimes get, there's a tokenizer step there that sometimes doesn't do whatever you want. Like, there's like a, I don't know, like if you know about the people watching, but you have like a tokenizer, but it's also batching stuff together. So like sequences have now the same length and now we have more complexity. Like Axolotl does a lot of magic for you. It does multipacks. So we put multiple sequences together and like then batch them so you can feed the GPUs efficiently. So yeah, a lot of things could go wrong there. Yeah. So decode your batch and check what you're feeding them all.
How did you concretely with Axolotl, you said like, okay, you were witnessing a training run. You saw, I guess you saw a loss, spike in loss. And then how did you correlate that batch with that loss with this tooling? Like how did you—
It's not that easy. And it's not that easy. Yeah, it's not that complicated, but it's not as easy as— I actually, you have to patch the trainer, the Hugging Face trainer that runs the training loop on Axolotl. So I patched, I replaced this step method. And then I was checking the grad norm and the loss. And whenever it spiked, I will decode the batch and log it to a Weights Biases table. So I could like see the step and then see the table. And I will save a checkpoint of them all just before. Because what I wanted to do there was skipping that batch because, okay, that batch is bad. I don't want to update the weights of them all. I want to just compute the loss. It's bad. I don't update the weights. So you skip the optimizer step. But like in a lower level library, like, you know, using like Axolotl on top of Trainer, having access to that loop is complicated. It's not as— so yeah, you need to take the library and actually override that part. I encourage people to try that kind of things and you could print them on the terminal. You could log them. The cool thing is that if you log them to something like a table here, you will have the run and the step and you will have the table that has like, I put the name on the table, like the step where it was logged and the GPU where it was logged. So it was like GPU 7, I don't know, sample 2 was having that spike on the loss. So you could actually log the loss separately per item on the batch instead of like computing the mean. And that's something also that could be cool to do. Of course, if you use like a lower-level library like TorchTune or pure PyTorch, it would be way easier. And I suppose like big companies or big players like Mistral, OpenAI, like log tons of proxy metrics. From all the nodes and all the different stuff they are computing. And so they can spot these issues quickly, restart, and like not lose money, basically. Yeah.
So, good.
Okay.
Some more questions.
If you just continuously log all this data, whatever, is there a storage cost of Weights and Biases?
Like, yeah. It'll be a storage cost.
Okay. Yeah, I don't know what that is. Is it expensive?
I don't know the pricing. I think it's fair value for what you're getting. Um, it depends, like, it scales on usage. So if you're using like personal, it's going to be cheaper. You get like a corporate license with all the security features, um, deployed on your cloud, yeah, it's going to be more expensive.
But there's a cost for the storage?
Yeah, yeah, yeah, yeah, there is. Um, but But you could use, like, you could use, you are not forced to use our storage. You could bring your own, like, buckets. If you put your Google Cloud buckets or your Amazon S3 buckets or Azure buckets, you can just reference the artifacts to the bucket you are storing the data. And you still get the lineage and, like, the complete traceability of what experiment used what data.
Can you tell me a little bit about the syntax here? Like, a lot of people open Weights and Biases, they get confused. Like, okay, you're showing that table right now, like the one you were just showing. It's kind of like, sort of looks like a DataFrame. Why do we even show the syntax here like that? Like, do you ever edit the syntax? are like—
Yeah, this is like a raw table. So it's like what actually gets dumped by the integration here. But probably, yeah, when you want to explore this, probably you don't care about all these columns and you want to, yeah, see the raw prediction maybe next to the data. So probably you're going to like remove some of those columns.
Oh, what I'm thinking is like, you know, on the top it says runs.summary and then— yeah, so like Can you like filter the columns by changing that or something? Or what is the idea there?
Yeah, probably. I don't use it. Like, I actually, I use the filter here or, or like click the columns. Yeah, there's like a specific language to actually query the tables here, but yeah, you will need to learn that. I think the same. Fair enough.
I was just curious. Okay.
So yeah, I think it's better like using the UI. That's not that— so yeah, you're— maybe you can group by here and let's see how it groups. Like, yeah, we automatically try to get relevant stuff here, but yeah, the group by doesn't make that much sense here. And reset the table. Yeah, I know, but here are basically to explore the individual samples of what we log. So as you said, it's like a DataFrame on the UI. So yeah, this is MMLU, had different pieces of benchmarks. So yeah, we have a bunch of tables here, 67 to be exact. If you want to dig, dig, dive, deep dive on here, it's— and probably you should not turn on every single model because what's happening under the hood is we're like concatenating all the data from all the results. You probably want to explore these tables like one by one for one model at a time and not all the models at the same time. So yeah, I think this is kind of a speed run of the way it summarizes models, features, and experiment tracking, what everyone, most of the people know here. So yeah, this is kind of the quick wins I show you. Like you can save your workspace view so you can organize your data, save it, put it in a name like this is my eval view, this is my training view. You have the run compare that quickly you can see what's different. You can pin columns to bring them to the left. Mostly if you are tweaking learning rate, batch size, probably you want to pin those so they're always visible. And there's more charts you can explore. There's the parallel coordinate plot. I think there is one here I quickly show with the beautiful lines that lets you explore. If you're doing hyperparameter optimization, it's very useful. So yeah, we can say that reproducibility in machine learning is key and it requires some effort. but there are quite benefits of using a tool like this and having reproducibility built in. Yeah, when for me at least—
I don't know if it means that your screen is shifted left or something, like it's not within the view, or is it just me, Dan? Can you see his whole slides?
Yeah, I think it's just you. It looks normal to me.
Okay, sorry.
Okay. Oh, so yeah. So no, for me, yeah, this is like a standard point of reproducibility. But what I mean is for me, the most important one is like, I can come back 6 months later and protect against myself and actually be able to, to re-inspect my project and maybe continue working on that or, or handling my project to someone else so they can keep, keep adding like more experiments or understanding what I was doing. So yeah, this is like a nice illustration from Christy Whitaker. I encourage you to check it. It's, yeah, it's a presentation that's online. You can check it out. So it's step by step, like using, yeah, versioning your code would be a step for reproducibility. Maybe next step would be like putting everything into a container, maybe using tools like Weights Biases to keep track of your experiments or organizing this, like when you start collaborating with people. So there's like multiple steps and like there's no 0 and 100%. It's like you can, you can be in between depending on how much complexity you want to add to your project. So yeah, that's, that's the takeout here. So there are different tools and Braze tooling that helps you accelerate your experimentation and come back in time and be able to rerun your experiments. So just to quickly Booth perspective here. What I show mostly is what we call now models. It's what you know from Weights Biases, the experiment tracking part. So I want to talk a little bit about like, we have a new product that it's tailored for like, okay, you have fine-tuned them all now and you know you want to use that model. So you want to put them all into usage internally or externally and you want to keep track of everything that's getting in and out of that model. So we are trying, we're excited to like show you here our new public preview product called Weave that, yeah, serves that kind of persona that's playing with the models and needs actually tracing and evaluation. So as Hamel Husain, like this table, sometimes like very clumsy when you are like running and you want to see inputs and outputs side by side. And yeah, and you are using a lot of models on API, like you're not always fine-tuning. So this new tool serves this kind of persona. So before diving into this, I'm going to just tell you that, yeah, this is another Python package. So it's nothing fancy. It's pip install Weavv and you get the package installed. It's a lightweight library that's built on top of Pydantic. It's very easy to use. And it's actually, when you start using models on API, you will probably have code around your API call. It's not going to be like, call OpenAI and get me the raw result. So you're probably going to have some Python code to preprocess, What do you mean it's on top of Pydantic?
Like, can you tell me what that— I'm gonna—
you will get it in a couple of minutes. So we leverage Pydantic. So this tool enables you to keep track of the Python functions you already built to make those calls externally. So you just need to add a decorator. So import weave, weave.op, and we will keep track of that function. So, yeah, we use this decorator to read the signature function and, yeah, and see what the model expects as an input here and what's the output. And we also have integrations with the popular vendors like OpenAI, so we also keep track of the underlying OpenAI call that gets called inside this function. So before just diving into the library, I'm gonna just propose you to, to, that we try to play with with a words problem that we may try to solve with an LLM. I, I had a lot of fun playing with this a couple of weeks ago, and I, I thought, okay, Wordle was fun, but the New York Times has multiple problems. And I don't know if you have played this, this word called Connections. So it's like 16 words and you have to find 4 groups of 4. This looks like a problem tailored for an LLM, or at least that an LLM should be capable of doing. We could maybe add this as an eval to whatever eval harness or something. But it's not as easy as it looks like. So yeah, these 4 words form a group. There's a unique solution. So there's 4 groups of 4 words. And you have to think that maybe if you put that word on that group, you will not be able to complete the next group because that word also belongs to the other group. So you have to find relationships between words. That's kind of an easier relationship. But yeah, and the problem actually, when, when you fail one of those words, It gives you like 3 mistakes and it tells you like you are 1 away. So one of these 4 words, even if they look like cow-related words, doesn't belong to the group. And yeah, that's the final solution to this problem. This was like 1 week ago. And as you can see, like the last purple one is pretty hard. Like it's bull, flea, meat, stock. There's those markets and it's not as easy. Like you need the to think outside of the box and maybe a person that is well-versed in English. I'm not a native English speaker, so I have a hard time with these problems, but I have seen some people that are really, really good and can actually solve this 100% of the time. I can't. I always use my 3 trials and most of the time I failed. So I'm also pretty impatient. So we try and like get something that works. So yeah, how would you solve this with, I don't know, OpenAI GPT-4? You may create a prompt that explains the puzzle and yeah, and maybe give the list of words and, and just ask them all like, hey, provide me a solution for this. Maybe you are better prompting. Maybe you put like some examples like, hey, a connection types of fish is a connection of bass, flounder, salmon, trout. And maybe you give him some hints like few shots of harder examples like that fire, things that start with fire. but may not, that may not help for that specific, like, 16 list of words you gave here. So if we try that prompt and it gets you something like this, so it gets one group correct. The anagrams, that's not as easy, but I was pretty impressed that this is a good starting solution, actually. And the others are like 3 out of 4, and I mark in red the word that doesn't belong to that group. So you could like iteratively solve the problem, like you could, try one prompt, maybe like point him out like, hey, this group have 3 out of 4, think outside of the box, maybe use Atisaurus to actually call and get the meanings of the words. But like, actually what I wanted to solve with an LLM here was the hard version of the problem. And the hard version is give me the full solution at once, not giving me hints of like, hey, I'm 3 out of 4. And As someone pointed out, like a good person that speaks good English, has good literature knowledge should be able to, yeah, find the solution at once. Maybe on a piece of paper, you write multiple combinations, but you find the solution at once. So we can use the LLM as a scratchpad. That means that we can call the LLM multiple times and maybe create intermediate solutions and then use another LLM or another Python code, because most of the time when you build apps, you have like crew, yeah, functions or external connections, maybe a RAG. Actually, you could try embedding the words and searching on embedding space and maybe finding a relationship of the words on that in space and maybe that proposing a solution. I'm curious if that works actually. And then finally, just submit, like, so you can run whatever you want, but you need to submit the solution at once and it will be good or bad, like wrong or correct. So the idea here was trying to solve the problem together. Colab was very finicky a couple of minutes ago, but we can try anyway. But so we, how do you solve a problem like this? I think Dan and Hamel Husain have showed this extensively. You need to create an evaluation dataset. So evaluation, and then you run your model against that evaluation. You compute a score in this case, like how many words correct, how many groups are correct on the 16. 4, 3, 1. 4 is perfect. 3 is almost, and yeah, 0 is none. And we put together this dashboard that we can, you can log and we can see if someone finds a good solution. You are not, it's an open project. You can, you can see and inspect what people are doing. So I wanted to show you this. So I built a Colab. So if that's the link, wandb.me/connections. And it's a very simple collab, so it's going to try to solve this as a very naive approach. So basically the prompt I show you on the slide, and yeah, it requires only Weavon and OpenAI. So if you have your OpenAI credits, that's great. You can use whatever model you have available. So as I said, like, ideally don't check here, that will log the results to the public project so we can have your results. So maybe iterate a little bit, play with this on your own, like, account so you can mess and maybe create a better prompt, maybe iterate multiple prompts. And then at the end, when you already have a good solution, maybe you toggle that on and you push the eval results to the leaderboard so we can inspect what you're doing. And basically to use Weav, our project to trace, our product to trace, you need to import Weav. This connects to Weights and Biases and tells him Okay, you need to log all my function calls to that project. So we are logging to the connections project. The first time you use Weights Biases, you will be required to paste your API key. So yeah, basically you click that link, probably if you have used Weights Biases. Yeah, I have SSO, so it asks me for that every time. Now forgets about this.
So you know, we're already past time, so just letting you know. In case you need to know.
Yeah. So yeah, key. Let me download a dataset here. You can run this on your time. So yeah, the dataset is like the list of words with the actual groups. You have an OpenAI key. So we use that. So as I showed you before, this is what you will do. Like you will have a call OpenAI and you will put that decorator on top of that. And that will be able to, if we call this function, you get a link as you always get with Weights and Biases. You get a link to the actual code. So it error out because, yeah, this is a nice feature of OpenAI that you can expect like a JSON format output. So I'm asking for the capital letter France in JSON format here. It will fix that. We will use the JSON output afterwards. So what, what do you get with that link? When you click that link, if you click that link here, you're redirected to Weights Biases Weave dashboard. So I have played with this project extensively. So you see that callOpenAI, the function we defined here, all underlying calls, like the actual OpenAI chat completion. So we know that, yeah, we can expand here and see, okay, we call with that list of messages, we call GPT-4 and the max tokens parameters we pass, and we got an output structure on JSON format here. So ideally what we want to do here is actually solve the problem. So we are going to just define a function to to parse the JSON we recover because the JSON looks nice. And we can actually put everything together in something we call a model that's a slight wrapper around a PyDantic base model. So it's just to keep everything organized. And it expects you to define just a predict function that knows like how to compute, so how to call the model. So this is the one-shot model, and we're going to use the same prompt I show you here. And if we call the model on those words, model predict here, we'll get also a trace to that specific, and we, it produces a solution. So if we click here, we get the, the model output on the predict function. So this function called the generate solution and that function underlying called the, so this is like a debugger. So you have like the traces of all the functions you are decorating and how the functions get nested. So when one of the functions fails, you can see what the inputs and outputs were. And click here to expand everything. And so we have the prompt, the comm, and everything gets version. So we have the one-shot model here that got version. So v0, the first version we have created of the one-shot model. So as I said before, probably you need to create like a dataset. In this case, we have a dataset and you need to create a function to check the solution. So here I'm just checking that the groups match. So check the solution. This solution is perfect. So the model, this is an easy, not every day is as easy, but this one was easy. And the solution was correct. We can click also on the link and we see that the check function, yeah, got a perfect match and we can expand and maybe compare manually if the match is correct. And to actually submit here an evaluation, you can run this. So this is a wrapper that basically says you are going to run the dataset against the model. So we're going to run the model against that dataset and we're going to score with the scoring function we define. Yeah, this is an async function because we run all the calls in parallel. And here we got like evaluated 20 examples and we, yeah, we averaged the scoring function. In this case, we got like 3 correct out of 20, not that great. So if you can beat that, it would be really cool. And if you click on that link, you are on an evaluation dashboard and this is the original dashboard I show you. So we have like a small leaderboard here and we see the actual true fraction here, 15%. I got only 3 out of 20. And I can have versions of my evaluations of the models I have been using. As you know, as you see here, I have been playing with this for a while. I have multiple models and yeah, I encourage you to try it and maybe submit something here if you want. We can dive here and see how the model is. As you were asking there, Hamel Husain, about how the model performs on every sample. So we see like every row here is a sample. So here the match is incorrect. We can dive here. So the one-shot model failed and the generate solution, we can inspect the solution and compare against the groups it generated and why the check solution was wrong. It got just 2 out of 4. So it's very handy when you start creating stuff that gets more and more complicated and calls external functions or maybe even external APIs. So I want to just finish with a slide, give it a try. The Colab should run. And if you have any questions, just ping me on Discord. I think I ran out of time, so that was pretty speedy. So if you have questions, or we have time for a couple of questions.
Ayush has been answering all the Q&A pretty, pretty well. I would say since we're out of time, let's head over to the Discord and kind of pre-audit there.
Yeah, there's some questions about getting code, but those are the sort of thing I think it'd be nice not to answer on the spot where it'll get lost. Anything you write in the Discord, people will have access to for a long time.
Okay, cool. Thank you guys.
All right. Thanks so much.
See ya. Ciao. Bye.
