Building with Instruction-Tuned LLMs: A Step-by-Step Guide

Foreign.

Hi everyone and welcome to the TO Building with Instruction Tuned LLMs, a step by step guide. We appreciate you taking the time to join us for today's event. We're so glad to see you tuning in from all over the world. If it's late, thanks for staying up to join us today. During our event you'll learn how to differentiate between instruction tuning and fine tuning of LLMs. We'll see how to use the latest models, data sets and training tools to both perform instruct tuning ourselves and and to pick up instructune models off the shelf to build powerful LLM applications designed for specific tasks through data centric fine tuning techniques. We're going to get right into it today. If you hear anything during our event that prompts a question, then please follow the slido link in the description box on the YouTube page and put yours in. Also, upvote any that you see in there that you'd like to see answered at the end of the presentation. My name is Greg Lochnane and I'm the Head of Product and curriculum at 4th Brain. I'm excited to welcome my friend and colleague Chris Alexiak to the stage today as we'll be working as a team to deliver this lesson. During the day Chris is the founding machine Learning Engineer at Ox Getox AI and by night he works as a fellow instructor and curriculum developer at 4th Brain. He's also a solo YouTube creator who builds weekly with the latest and greatest generative AI and LLM tools as you'll see him do today. One quick note on the flow of today's event, I'll be sharing slides and high level concepts as an introduction to coding demos that Chris will run. But first we'd like to get started by sharing a motivating example together. Instruction Identify the odd one out and explain your choice. Input Orange Green Airplane Chris, can you walk us through the responses that we get here?

Yeah, you bet. So the first half year we have our non instructune, that's our base model. Its response is Orange is the odd one out. And the explanation is that orange is the odd one out because it's the only one that is not a plane, which is kind of nonsensical and doesn't really help us at all. Then for our instruction tune model we have a much better response which is Response Airplane is the odd one out. And then it gives some cogent kind of explanation here. Airplane has nothing to do with the color spectrum. It uses aerodynamics and other techniques to fly. It is not on the ground and it's made out of metal and durable materials. So it's. The improvement is pretty clear between the two.

Substantial improvement on this and many other questions that we can get with instruct tuning. Super cool to see this. We're going to see lots of examples when we see you back in the demo. That's coming up in just a few minutes. Chris, see you soon. So today I want to provide a little bit of context before we jump right into the demos. We're going to talk about the big picture of LLMs and the big picture of fine tuning versus instruction tuning. Then we're going to see some instruction tuning. We're going to see how to leverage some open source state of the art models and data sets as well as fine tuning techniques. And then we'll also see how we can take an instructuned model off the shelf and then fine tune that model and really dial in the input output schema to make it very good at a specific task. Okay, so the context here is that if we look at the lineage of GPTs, if we look at OpenAI's GPT GPT 2, 3, all of the LLMs that we've seen, they're really built on this foundation of unsupervised pre training. They this is really leveraging a self supervised approach where there are no labels to the data and we're essentially scraping in a ton of information from the Internet. And beyond that unsupervised pre training first step, then we do some supervised fine tuning. That's what allows us to really dial in how good we are at these benchmarks. And the benchmarks are, are the classic NLP benchmarks. They're the question answering the information retrieval, the sentence completion. And what we've seen over time is that if we just use more compute and more data, we can do better and better and better at each benchmark that exists and even include more benchmarks and get better at those as well. So this is sort of the big picture of the LLM. Now when we go use the LLM, we want to make sure that it's good not at everything anymore, not at every benchmark, but at one specific thing. And that's where we get into this idea of prompting or prompt engineering. This idea of zero shot learning is simply writing a prompt and trying to get the LLM to respond to that prompt. As we saw in our motivating example, a few shot learning technique would be giving the LLM a few examples, maybe one, maybe several, that would all be under the guise of few shot learning. Then when we talk about fine tuning in this context, we're talking about fine tuning for a specific task this time. Maybe we have many examples, maybe it doesn't fit in the context window anymore. What we're really doing is we're changing the input output schema. It's no longer a model that does everything, but instead it's a model that does one thing. This is another way of thinking about getting started, building applications. Whether we're using zero shot or whether we're using few shot, we're really exploring that space of all possibilities within the LLM. This is cheap and this is quick, and this is a great way to just develop your mvp. Now, as you want to really explore that particular area of the LLM, the latent space within that area of the LLM, what you're doing is you're carving out a region where within the LLM that you want to work within. And so there's a lot of ways you might visualize that. I offer one here that was created with Mid Journey. So if you want to sort of imagine that you're in this crazy space and you're trying to find one area of it that your application resides in, that's really what you're going to do when you're dialing things in with the fine tuning on top of the LLM. Okay, so just sort of wrapping up this idea of instruction tuning versus fine tuning. Before we get into it, we're really asking, what task do we want our LLM to have superpowers on? That's what this fine tuning of the input output schema when we're building apps is doing. Whereas when we're talking about supervised fine tuning done on large language models in general, that's more akin to the instruction tuning that we'll see today. Whereas instruction tuning is really a way of doing better, not just on the classic benchmarks, but on being aligned with humans, being aligned with what we expect when we give instructions or direction, and importantly, being aligned when it comes to bias, truthfulness, toxicity, and other metrics and evolving benchmarks that we see coming onto the scene all the time, we're really saying, okay, I want to develop my LLM so that it could be any superhero aligned with any task that I might want to help a human out with. That's what we're doing with this instruction tuning and then with the fine tuning of the input output schema. When we're taking that general and making it specific, we're now saying we want to to define how a user will interact with our application. And we want to make sure that it's really clear what that interaction is like and what that superpower and what that single task that we want to get really good at is. So in short, instruction tuning enhances supervised fine tuning. It's really a subset of all possible fine tuning focused on alignment with humans. And it's concerned with following instructions and then fine tuning. That input output schema is all about dialing things in for a specific task. We will see both today and first off, we're going to start with instruct tuning. Now, we picked a couple of pieces of technology that are particularly relevant today that we want to show you how to deal, how to work with. Dall E15K is a data set that was released with Dall E 2.0, the open source LLM. And it contains 15,000 high quality human databricks employee generated prompt response pairs. The structure here is instruction, context response, and each of these pieces of data falls into a different category of instruction. So you might see creative writing, closed or open question answering, summarization, and on and on. We see a quick brainstorming example of what to put in our PB and J. If we don't have any J, and these can also be used commercially, we can leverage Dall E15K, we can leverage Dali 2.0 for commercial purposes. What we're going to do is we're going to leverage Open Llama, which is a reproduction by OpenLM Research out of Berkeley that essentially recreates Meta's llama, which stands for the large language model Meta AI. As of May 22nd. Just a little over a week ago, they released a new checkpoint for the open llama 7 billion parameter model. They expect their full 1 trillion token training run to be done here pretty soon. This is trained on the red pajama data set. The full 1 trillion tokens is associated with that data set. And this again is a commercially available and can be used for commercial purposes model. We're going to use the Open Llama model in addition to the Dall E15K dataset. Interestingly, today what we're going to do is we're going to leverage a really new addition of how to do fine tuning very efficiently. And this is called Qlora. So this is taking sort of this big idea that is sort of an emerging best practice with LLMs that downstream tasks, meaning more specific tasks, have intrinsically lower dimensions than the higher upstream tasks. So all possible benchmark tasks, a lot of dimensions, one specific task, not very many dimensions. So when we're doing fine Tuning, we can get away with a lot less compute and we're going to leverage Bits and Bytes, a library we'll hear more about for quantization. We're going to leverage hugging faces parameter efficient tuning methods. And we're also going to note because we'll come back to this in the second demo that Q Laura, it was released on May 23rd, so just again a little over a week ago is actually an improvement over the LORA method, the low rank adaptation of large language models method for training. And so Chris is going to walk us through all of this today. He's going to talk about quantization, he's going to talk about how to think about each of these libraries and you're going to get a real feel for what's happening at the state of the art in code. So with that, we're going to drop the link in the chat for you to follow along as Chris shows us how it's done.

Chris hey, thanks Greg. Today we're going to be looking at exactly what Greg said, which is supervised instruct tuning. Open Llama using the Dall E15K data set, we're going to leverage a few very powerful libraries to get this done effectively and efficiently. The whole fine tuning process can actually be run in Colab with Colab Pro. That's the amazing power of especially Qlora. Let's just start going through this and see how we can instruct tune. As Greg said, instructuning is a subset of fine tuning. So we're using supervised fine tuning to let our model get better at instruction following tasks. First things first, we got to get our dependencies. As always, we will want to ensure that we're pulling right from the GitHubs for these dependencies since some of these libraries are pretty new and they're evolving very quickly. They're resolving bugs, they're adding new features. So pulling straight from the source is just going to be easiest to make sure we have the most up to date version. In order to get our model to get better at instruction following, we have to show it a lot of instructions and we've chosen to show it instructions from the Dall E15K data set. Now, this data set is full of 15,000 instructions. They have three possible fields of a context, a instruction and a response. They also have a category column which we won't be leveraging today. But the idea is that this method of fine tuning for instructions works better if we let the model see many different varieties of tasks. And so the category shows us tells us which task that particular row belongs to. Our instruction is straightforward enough. It says what to do, context is an optional field, and then our response is the output. There are 15,000 rows in this data set. However, some of them are.