Fine Tune a LLM with Low-Code
Ray Deck walks through fine-tuning large language models using low-code platforms like Hugging Face and Replicate. He covers data prep, quantisation, and deployment without needing a PhD, then answers questions on model selection and private inference.
Chapters
Thank you everyone. I'm so flattered that you would share part of your afternoon with me. My name is Ray Deck. I have sort of a weird background in software. I came in as a political science major. I was actually being trained as a data scientist. I just didn't actually know it. I would later go and work in software startups and always had the perspective as I got more and more into the tech side of the business really looking at from the data point of view because of how I was trained by Tufti and others. And that perspective has, I think, set me up well to have a greater, at least, appreciation of the technological revolution we've been going through in the course of the last, I would say, about 12 years since our friends in Canada first came out with AlexNet, which sort of restarted AI back in like 2006. 2011, I think, was the paper. And then going through more and more and more turns on this, first with machine vision in the, you know, in the latter part of the previous decade. And then of course, as we all know, the generative AI and really natural language revolution that we've been seeing in the course of the last few years. I currently run a community called StateChange where I'm teaching a lot of product owners and other people to be using no-code, low-code tools to be able to make for them relatively advanced applications and tools to create economic value for their businesses. And this also gives me a weird perspective because I have the idea that you don't need to be a PhD in order to be able to really gain mastery over these tools and these technologies. These people say, well, I'm not a computer scientist, I can't really make software. I think we all know better. And same thing I think applies when we get into the AI. Part of what I hope to do today is give you that kind of comfort that even if you don't know exactly how you're going to do all the pieces, that these are things that you can work with and that you can customize and modify and make your own. I think when we're in those, in that position, we're able to do bigger things with these technologies. I also want to give a quick credit to my research partner in this. We've been meeting about once an hour over the course of the last 6 months first doing hackathon work and then more recently doing just research on these various AI technologies, including a lot of things I'll be talking about today in terms of fine-tuning and training and what have you. I find all this is much better with friends, and so I'm glad to have made some in you all today. And I would encourage you, if you're going to be investigating more of this, to be doing this, you know, with somebody else. It really helps keep going. And actually, by the way, it's just a a brilliant engineer currently working for you. And I did not make this name up. It's actually called the Canada Startup Company up in Toronto. So I took a poll before asking how many people use ChatGPT. A lot of you have. I think that's the way a lot of us have been experiencing the current generation of generative, what are called large language model technologies. And ChatGPT is really powerful. But it is a service that's sort of a little bit higher level, sort of mixing together a few different ideas. It's also not the only way of doing these things. That while certainly, I mean, OpenAI has like an $80 billion valuation, and of course certain political drama that we've all been hearing about lately, there's actually a big universe of other models and other technologies out there that we can start to play with. Now, the idea behind what we call a large language model really came up in a paper from 2017 called Attention is All You Need. And that one in turn was built on ideas from what we call natural language processing and natural language understanding, which is based on the even deeper idea that a word's just a number. Like if you took all the words in a dictionary, and you said, well, this is the first word, this is the second word, this is the third word, they're all just words. They're all just numbers, right? And so that takes away some of the magic associated with these large language models that you can say, well, instead of like, how does it know what the next word is? I just figure out what the next number is. And does some other techniques to figure out how to construct numbers and new parts of speech and all that. It's called lemmatization. We can talk about that more if you're curious. But as a result of all being numbers, it's all just statistics. And because we're just talking about a certain type of applied math, these are things that are very easy for machines to do. And then you can take it up another level and the, the, the, because of the nature of the kinds of math that's being asked to do, it's mostly matrix multiplication, which is actually relatively simple math as these things go and just rewards having a lot more cores. Right? You're asking it to do lots and lots and lots of processes, most of which are not dependent on each other so you can parallelize them. Now your basic CPU is really powerful, can do these big complex operations, but the insight that came up like, I don't know, almost 20 years ago I think, was that you didn't need all of those complex operations for a lot of very specific use cases, particularly computer graphics. And that leads to why GPUs work differently from CPUs. GPUs do a few complicated things very fast, but like only a few at a time. Whereas GPU does many, many simple things, largely for the purpose of drawing triangles on your screen, many, many times. And because that kind of math, being able to do lots and lots of that, was a big insight that came up, I think this is sort of the AlexNet insight from 2011, other than we can then apply not only computer graphics, but also to words once we're just turning the words themselves into numbers. The two big ideas are first the idea of attention, which is basically that like, you know, you could kind of put numbers against the position of each of these words, knowing what the third word is, knowing what the fourth word is in like a given document. That's kind of a high-level gross reason why attention is all you need mattered for being able to make the large language models that we now see work, plus the computer horsepower to be able to do lots and lots of calculations. That's why we can have these really big models that have lots of calculations it needs to do. And then most recently, a thing that we're going to talk about a little more today is something called LoRA, which is an approach to fine-tune natural language models. Any of you who might have been to like a session on fine-tuning, say 6 or 7 years ago, would have heard that you can fine-tune a visual model, right, like something that figures out dogs versus cats, but you can't fine-tune or can't do transfer learning, as we call it, on a natural language model. But now we can. Now exactly how that works I think might be a little beyond the scope of what's interesting in this conversation, but the fact that we can means a couple of things. We don't need to rely just on OpenAI or just on GPT. There's been this like explosion of all these different models. I know it's like tiny text and you can look at it on the download version of the slides. But the key thing is there's just a lot of them. Some of them are by big companies who have spent hundreds of millions of dollars to go train up a model. Some of them are research projects. Some of them are very closed source and some of them are open source. And like, we wind up with— I mean, of course there's OpenAI out there, but there's also some others that are closed source or services run by companies like Cohere and Claude by Anthropic. Anthropic actually is an offshoot of OpenAI. And then there are some others that are really neat, like PaLM, which is the one by Google. And then you have some that are open source, and that means you can download them. Anytime you want. They're just huge. The best place, the best way to sort of get started with like looking at that explosion of models to go beyond just ChatGPT, I find just go explore. There's like 3, I think, pretty good places to explore that I recommend you do. One is called replicate.com. Replicate is really nice because it gives you a serverless interface to be able to run all of these models. I was asking before about like who might have been using this as an API, and the— you can go click into all these models and it'll just tell you what its inputs are, i.e., what question you're asking, or hey, here's the image you want to start with, or whatever. And then you can go run the model. And these models are so weird. I mean, some of them are going to be like the kinds of question and answer stuff you've done before, like with the ChatGPT, but some of them are going to be things like It's going to go make a song for you based on a prompt or based on an image. Or one of my favorites, actually one that actually I think this came out of NVIDIA recently. It makes 3D meshes, right, that you can put into Blender or into a VR context or what have you based on the inputs, right? I asked for a World's Greatest Dad mug. It gave me a mug that said something on it. And so there are varying levels of quality. Right? But one of the really neat things about it is that most of them are open source, and you can then go over to a certain place called Hugging Face. And Hugging Face provides a couple different services, but really its biggest role in the world, I think, right now is this kind of like the GitHub for AI. People can upload not only the code related to these models— like, we usually think of code as like, you know, your Python code or your JavaScript code whatever else it might be. But whenever we're dealing with AI, you're actually dealing with what we call weights and biases, which are, which essentially, that like, you have not just the immediate formula, but also the, it's basically like a giant pachinko machine, right? You're putting your data in at the top, it's hitting all of these little operations along the way, and you're using a little bit of code to figure out what all of those operations need to be that add up to generally moving in the direction you want it to. And so like there's a little bit of code but a whole lot of these data files. And these data files for the tiniest models, or actually all these data files are roughly proportional to what they call the number of parameters that are inside one of these models. For perspective, ChatGPT GPT, what was called GPT Turbo 3.5, was a roughly 20 billion parameter model. So about 20 billion numbers in it. ChatGPT, we believe roughly that GPT-4, of course they don't publish these numbers, right? But we think it's well north of 200 billion. There are other models out there that are, some of which are as small as like 2.7 billion, some which you might call more mid-sized, maybe 7 billion or 14 billion, and there are a few, even in open source, that are as big as 180 billion. Now the reason I share all those numbers with you is because each of those parameters is just a number. In fact, it's two numbers, weight and bias. So when you ask how large is this model, it's going to be a question of how much How many bytes does it take to store a number? So if on average it's taking you about 32 bytes, sorry, 32 bits, 4 bytes to store one number, and it's 175 billion parameters, you can just sort of do out the multiplication, right? And what you'll find is when you go over at Hugging Face, you can click in here and go into, like, for any given one of these models, click where it says files, and all the files are there for you to download. They're just enormous. In fact, they're usually sliced up so that it's about 4 gigs per chunk. And so like the— it might have one chunk for a, you know, 2.7 parameter, a billion parameter model. And you might have like as many as, well, I think it was 80 chunks when I checked the Falcon 180 billion parameter model before we came into the session here. And then there's one more I'll clarify that is kind of interesting and different because it not only includes includes open source models like Hugging Face does, but also has a bunch of closed source models, provides you with APIs over to those. I find, and I would encourage you, if you do nothing else with the end of this talk, go, go— nope, I don't want to leave Canva. Why would you do that to me? Okay. Go check out, play with one of these. Play with it over on Replicate. They give you free credits to go try it out a little bit, and then you can try it out a little bit more by putting a little bit of money into the machine. You will be surprised at like the different and weird things you can do and start to get ideas about what more you can bring to the world. What's Replicate again? Sure, replicate.com is this site. And what they do under the covers is they are storing these models and serving them up essentially in Docker containers and serverless containers so that you can click on one of these or reference it by a URL. And run this model. On the website? You can run on the website and you can run over an API so you can just go directly into your website or your application or what have you. So they are an API as a service, kind of like RapidAPI maybe if you've ever used that for other third-party services. But they provide all these both on the website as well as APIs so that you can, well, have them do whatever it is you want them to to do, right? Which could be generating a 3D mesh, it could be writing a song, it could be answering a question, it could be whatever the individual model is supposed to do. Does that help? Yeah, yeah, yeah, cool. It's kind of like a bunch of types of ChatGPTs on a website. Yeah, I mean, the interface is not nearly as good as a ChatGPT, right? It's, it's very much one-off. Okay, you're running it as if you're just running that one API call, and then if you want to create more of a ChatGPT experience, that's going to be the thing you sort of stack on top of it. Juan Pablo gave a talk about that a little bit earlier today through the Microsoft tools for doing that. I think the gentleman next door is talking about that from a UX point of view as well. So we sort of talked about having a world of open source and closed source. And this is actually something that Dolly 3 made for me, talking about like what are fun things you can do in a public space versus what are things you can do when space is private. And I think that's a useful framework for thinking about this a little a little bit, is what are the kinds of utility you're going to get from a ChatGPT or from a GPT-4, like what OpenAI or what Anthropic is doing with Claude or what have you, versus what you can do with the open source models? Because certainly there are plenty that fit in each category. So this is LLaMA, which is actually made by Facebook, right? But Facebook's strategy, when they sort of were called out by OpenAI, Instead of saying, we're going to make a better one, they said we're going to open source it. I thought that was very clever of them. I don't know how long they'll keep doing that, but certainly LLaMAs are some of the state-of-the-art models. There's a company called Mosaic that recently got bought. Mistral and Falcon are both really interesting open source models too that actually get very large. I think I alluded to the 180 billion parameter Mistral model, which is a large language model. And then on the closed source side, there's more than just OpenAI. OpenAI. Anthropic, Cohere, AI21 are all major players in the large language model space. Many of them are cheaper and they have some interesting additional features, like AI21 does better retrieval augmented search, which can be pretty cool. And I actually just ran across this company from South Korea called TwelveLabs, which was invested by some senior— some people from Google who I really respect. Which is actually about analyzing video, right? So it's all video first. So there are lots of them out there, right? But they're kind of— but they have different attributes and they're good for different things. So like the open source models you'll find in general are a bit smaller, as opposed to closed source models that are generally a bit bigger. And this is sort of intuitive because the closed source models are super expensive to build. If it were cheap to build, they couldn't make money on it, and that's why it goes more into the open source model. The developer experience of these things— actually, I think Replicate makes the developer experience of working with open source models much better. That's why I'm recommending you to it, so you're not sort of scared off by it. But certainly the people over at, like, particularly OpenAI really know what they're doing to make it easy for you to work with. But now we get to, I think, a key difference from the point of view of our discussion here today. Which is how to gain ownership and make these things really your own. Which are the issue of the context window and the issue of hackability. The context window is kind of a big idea. Like, I think you've all heard already from other people so far that like the job of a large language model is to figure out what the next word's going to be, right? Or as I put it, what the next number's going to be. But how does it know? It's gonna know based on two things. Actually, you and I were talking about open book versus closed book tests, right? I think that's a good analogy. So there's like, if I just ask you a question, right, why is the sky blue? You're working off your memory, right? And that you're dependent on whatever was in your brain as of the point I asked the question, and then you're gonna start answering based on your best knowledge. The The opportunity is both if you have lots of memory, well, maybe you know more, right? What if you never learned why the sky is blue? What if you don't actually have that knowledge with you? What if I could give you a cheat sheet? What if I gave you, here are 100 fun facts about physics? It'd be a lot easier for you to answer the question at that point, right? Like, huh, that seems like an interesting fact, I'm going to go look at my chart. So the context— the idea behind the context window is instead of me saying, "Why is the sky blue?" I can say things like, "Why is the sky blue? And by the way, here are 100 fun facts about physics." Right? That when I give you that information plus giving you the question, you can give me a different quality of answer than you could before. So one of the things that you'll find that like— did anyone here watch OpenAI's Dev Day a couple weeks ago? One of their big announcements was that the new version of GPT-4 has a context window of about 120,000 tokens. That's about 90,000 words. That's a couple of books. And the reason why that's significant is that instead of me having to give you a cheat sheet, I can give you a physics textbook, right? And that physics textbook can go into that context. So even though you haven't studied physics, You have a physics textbook next to you, and now I'm going to ask you why is the sky blue? Because you have that greater amount of information, there's a much greater likelihood that you're able to answer the question. And so this is why the context window can matter. It's like how much can you augment their knowledge at the point you're asking the question with additional data like an open book test. And then there's the issue of hackability. Now you can do fine-tuning for the closed source models. It is, as Altman pointed out before, before he lost his job, are very expensive. But the open source models you can tune till the cows come home and you don't need to pay them, don't need to pay whoever made the model for that. You will need to pay in terms of compute, we'll talk about that in a minute. And then there's a question like, how much does this stuff cost? I will tell you that in general if you haven't worked with any AI before, the closed source models are going to be much cheaper., and they're only charging per invocation. But depending on what answers you're really trying to get to, it can make more sense to be asking the same questions of an open source model that's fine-tuned as opposed to a closed source model. But overall, I think that the big reason to be going one way or the other is going to be because of what your other, other needs are in terms of how you're trying to customize or create different kind of experience. So These are, to my mind, 6 ways that you can make your own large language model or make it your own. The first one is you can make your own foundation model, right? This is what those other open source guys I was talking about do. This is what OpenAI did, obviously, etc. It is a lot of work. It is extremely expensive. I don't recommend it out of the gate. But the other ones are really cheap to do. And I would— and like, if you're going to be thinking structurally about this, this I think is one of the one of the key ideas I have here today, is the idea that you can do fine-tuning, which is to transfer knowledge from, say, a book into the model. So basically, like, it's like continuing to run to build the model afterwards. This is what that LoRA stuff I was talking about before is. It's a cheap way to say, look, you've built most of your brain, and now we're going to add a little bit more. To use an analogy, instead of being an open-book test, this is just getting it to study for the test. And this is actually how a lot of models have been trained to take tests, right? To actually do the sort of question and answer format is actually by training it to answer questions to learn how to do that kind of structure in addition to learning about the domains. Data augmentation, this is the open book model. This is often referred to as RAG, but I wouldn't put anything too special on it. The idea is you get more information that you think is going that are going to be relevant to the user or to the question at hand, and you're basically putting it right into that transaction, right into that prompt. And then, you know, you can do a little bit more. This gentleman here before was actually asking me about what boils down to prompt engineering. How do you get a particular model to do the things that you really want it to do? And this can be something that is not super clear because you're dealing with the super complexities of something that has 180 billion different parameters. And the— say, you'd think, well, if I put in this, I should get out this. But when you have 180 billion parameters, there are lots of opportunities for it to take all sorts of weird little roads. And that's part of the big opportunity of it, but it also makes it a little bit less deterministic. And so learning about how— learning techniques for how we can get it to give us the answers we want matters. I will tell you that, like, if you look under the covers of a lot of prompts that are wrapped up in these services like ChatGPT or whatever, They often start with, "You are a helpful assistant." Like it needs to be told to be helpful. Sometimes it has to be told not to be evil. I saw another person who said, who pointed out that like it often helps like when you're asking complex questions to say, "Give me a correct answer." Otherwise it just wants to give you an answer. These are all examples of prompt engineering that you can put, bring to bear. To make these things work. There's actually a whole literature about this that looks a lot more like the psychology literature than it looks like computer science literature. There's something called continuity of thought. Continuity of thought is basically if you give it examples of questions and answers before you actually ask it the question you're about to ask, it tends to do a better job of answering the question you're looking for. This is just priming the pump, right? This is like giving it practice questions on a test first. There's another one called constitutional AI. Which is basically like giving it a set of ethics or how it should be thinking about the question, which again is like a paragraph of text you can put into your prompt. This is actually even more interesting in a world where you have sufficient context, right? Because the more context you have, the bigger the window, the more text you're allowed to put into your prompt, the more room you have for adding some of these other interesting considerations. Okay, but having done all this, then we've asked a question or put in input, we've got an output from the model, Then what do we do with it? And there are sort of two aspects to what we do with it. The first is the idea of you can feed it to another agent. Putting multiple agents together is a really powerful way to get more juice out of this. There's actually a really interesting Microsoft project called AutoGen that's been getting a lot of ink lately. And if you haven't used it before, I really recommend that you maybe check it out. The idea is sort of it's a party line of running multiple agents where the agents are just taking the answers from other agents and using it as inputs to themselves until they kind of come around to what they think is the right answer. It's like combining writers and editors, right? Like, hey, you write a thing, the editor says, well, I don't know, you really can do better with this, and the writer says, okay, here it is better this way, and the writer says, that's fine, you messed up this other part that previously was good, and they just kind of go back and forth. One of the neat things about agent composition is is that from the point of view being an AI company, is that makes you make more requests into the system. And so it tends to be much more expensive, right, than just asking one question. But actually under the covers, this is something like ChatGPT and like, you know, Azure and Copilot stuff are all doing for you in order to get dialed in what your, a better answer. And then finally, of course, it's like, what is it actually going to do for you? How do you turn this into an application? I talked to a VC a few months ago after a hackathon where he said that like the project that me and my partner had submitted was one of the very few that wasn't another chatbot. Chatbots are super popular right now, and certainly AI makes chatbots possible, but there are so many other applications you can bring to bear that create more economic value, even if they're not as cool a demo, right, that like the literature major might might appreciate. Keeping an eye on the ball of like what creates economic value is something I talk about all the time to my no-code and product management folks. And I think we should be keeping an eye on it as we think about AI as well. So when we think about these various steps, how do they differ in terms of using open-source models versus closed-source models? And really it kind of boils down to this, which I think is sort of Key idea: if you want to do retrieval augmented, you do that with a closed-source model. That's how you inject information. You can give it an open-book test because it knows how to do an open-book test. The open-source model usually doesn't have as much context window. It can't take as big a cheat sheet into the room, and as a result, it really profits if it actually studies for the test. So the best way to get information or to create a more custom experience for an open source model is going to be fine-tuning. And if you're looking to do it using retrieval, you're going to be better off using the closed source models that have larger windows, at least until such time as the open source models get either bigger windows or compute becomes a lot cheaper, as we'll talk about in just a minute. So let's leave retrieval augmented aside for a second because I think other people have talked about it today. But it is the thing that if you're just looking to do one thing to make it your own, I recommend you do it. It's a great place to start. But I heard when I was talking about fine-tuning, a friend who was also a fellow entrepreneur say, Well, Ray, fine-tuning, that's just for PhDs. It's not. It doesn't have to be. It used to be super, super hard to do. This thing with LoRAs made it much easier to. It doesn't take nearly as much code. And then that standardization of how it works has turned into some tooling that's much easier to work with. So the cost we want to consider are going to be, I mean, how much does it cost to learn, but then also what's going to cost us it's going to cost us to store the changes we've made and how much it costs to run. Remember I was talking about like how big these models are, like, you know, gigabytes of, you know, multiple tens or even hundreds of gigabytes of information. The storage is actually quite real compared to most other files you'd be thinking about like in GitHub. And the cost to run is also going to be non-trivial now that we're no longer just sitting on top of OpenAI's infrastructure. So there are a couple different ways I have seen for thinking about fine-tuning. I'm going to leave aside the high-code version of this. That I've seen for doing fine-tuning. OpenAI, which of course is a closed model, and AI21, which is also a closed model, both offer fine-tuning. It's very expensive. And Replicate, which offers a much cheaper version of fine-tuning because they basically give direct access to their GPUs. You're renting their time. Hugging Face has this as well with something called Auto Train, which is now Auto Train Advanced. I think originally Hugging Face Advanced was supposed to be the advanced or high-code way of doing it, but it turns out to be the one that works a lot better. So the original Auto Train, they actually deprecated everything, Auto Train Advanced now. So don't be too scared by the fact that it says Advanced. I'll talk more about Hugging Face Advanced in a second, but let's talk about how we set up our data. So the, um, we talk about this in terms of the models are always trying to reach for the next number, trying to reach for the next word. What the way that the first 3 places I was talking about, Replicate, that OpenAI, and AI21 work is that they all encourage using something called JSON-L. Anyone here worked with JSON-L before? Okay. Anyone here worked with JSON before? Okay. So JSON is the JavaScript Object Notation. It's a way of describing data that is largely compatible if you were to copy and paste it directly into a JavaScript file. You could say just const x equals this first line and JavaScript would know what it is. And that's kind of the idea behind JSON is it's a way of encoding data that is very compatible with the way JavaScript works. Now the problem with JSON is that I have to keep reading through this whole thing, get to the very end, before I understand how it works. I have to know everything to know anything. So this creates a problem. If I want to train based on 5 million examples, man, that's a lot of things I need to load into memory in order to deconstruct that, right? So JSON-L. L is for line. So the idea is that JSON— see where it says See the backslash n here? That's a newline character. It's been encrypted, it's been encoded into the JSON. That way JSON is always going to be on a single line. That's a guarantee. That's a contract for the way JSON works. So whenever I get to the first newline character, I know carriage return, I'm coming back to the beginning. Right? Works the way a CSV does. It works the way any other line-delimited file does. That way I get all the ease of describing key-value pairs, right, that I get from JSON, and I get the memory management, right, and the streamability of being able to think about one record at a time. And now instead of having to come up with some new structure, great, I've got this. I can do JSON, JSON, JSON. Once you— this is not something that's usually built into most models like Python or or, or JavaScript or what have you. So what you wind up doing is looping through and you're doing a JSON.stringify, add a newline character, JSON.stringify, add newline character, go generate the file that these folks are going to be able to use. I bring this up because I can't tell you how many people I've seen get totally stuck on the idea of fine-tuning their models. They're not sure how to get the data in there. The JSON-L doesn't need to be super complicated, and it really can look just like what we see up here. So the, um, the other side effect of having JSON is that of course you're just feeding text into the model, but like Replicate or AI21 or whatever, they understand how the model thinks about the separation of questions and answers, and they're just going to transform this JSON into the text that actually goes into their model so you don't have to think about it. This is definitely the way to play on easiest for creating a fine-tuned model, particularly if it's going to be bad text. Does this make sense? Any questions about this? CSVs work basically the same way. For those of you who don't know, CSV is a comma-separated values file where instead of being wrapped up in curly braces, it's just separated by commas, and usually big strings are wrapped up in quotation marks. I like this less, and I like it less because it does not have nearly as many standards, right, to guarantee that you're not getting multiple lines or getting— or find other ways to get into trouble, whereas JSON has a lot of standards that also mean that it sort of automatically gets taken care of for you in code. But you can sort of see that it works on the same principles, right, same key-value pairs. Oh, one advantage of CSV, just before I leave it, is shorter, right? You see how these lines are shorter than the lines I had in JSON-L? So if you're just worried about your overall file size, it's a good way to cut down by like 40%. Okay, so in both those cases, you create the JSON-L. You can either use a command line program or just directly upload it into, you know, OpenAI or AI21 or Replicate or whatever, uh, to get the, um, and it will just start a job and it will start creating things for you and it will then charges money. AutoTrain is an open source package, open source library that is largely consumed as a command line. So this is something you find a lot in Python, right? You install a pip package and you wind up with an executable and it just wants you to run that executable. Originally AutoTrain was none of these things. It was a service just like the one I was just describing. Dump in your data, we'll give you a model. Over at Hugging Face. They then created this advanced version and they deploy as a Gradio app. Do people know what Gradio is? Gradio is a framework for making super simple single-form applications. So like you just sort of, you know, hey, these are my parameters for the model I'm looking to edit, and then just kind of goes. It doesn't work that great. I find that like whenever I got into trouble, then there was nowhere to go. So what I would do instead is pip install. So I talk about things being, you know, no-code a lot with my clients. I will talk about this being low-code. This really does work best when we're in a local environment. And so now the question becomes, where do you want to go run this stuff? I mean, you can run it on Hugging Face, but I have found— I got this advice from Jeremy Howard, who's the, um, who who teaches a course. I learned it from his course. He's a founder of Kaggle from back when. And his recommendation was to use this thing called Paperspace, which was recently bought by DigitalOcean. The model essentially is they give you access to GPUs. They charge per hour. They have a really important feature: auto shutdown. I'll be mentioning auto shutdown again. This has bitten me more than once. The prices are pretty reasonable, I gotta say, on a per-hour basis. And they have these two sort of sub-services. One is for virtual machines that I use when I want to do some deeper things. And the other is Gradient, which basically gives you access to a Docker container. Right, so that you have your Docker container and it will spin up your Docker container and attach it to a GPU or GPU instance. And then you can use a notebook., right, like a Python notebook, or just use a shell and be able to do all the work you want to do in it, including be able to pull data in, shove data out, whatever. And then after you're done with it, you shut it down again and they stop billing you. Another very cool thing about it is you still have access to your files after you shut it down, right? Like if you've ever been like, had a machine like on DigitalOcean or something, the, you know, One of the things you find is you shut down the machine, you don't get access to the hard drive anymore, right? And if you— and also until you've destroyed the machine, they keep on billing you for it. That's not the case here, right? It's much more as long as the container is running, they bill you for it, and you can access the files. Those two things I found to be really important for making it easy, right, to get access to this stuff. So for my tuning, I use what they call the Gradient Notebooks. And I discovered that because AutoTrain works on the command line, instead of using the notebook, you know, the Jupyter Notebook interface, I went over and clicked on the terminal, right? And it was a couple lines of terminal code that were most important. And just to give you an idea of how much this stuff costs, I'm not sure how well you can all see this, but like they offer different levels of GPUs and CPUs. And yes, they do offer CPUs. I don't recommend that for training large language models. But they have like a B100 here, which is, you know, a few years old in terms of state of the art, for about $2.30 an hour. I'm going to just keep you by perspective because that's per hour, right? And that— and there are 168 hours a week times 4 in a month. You can see how this can become a very expensive habit if it does not also shut Now the— oh, one final thought here is I also picked one of the faster machines. This was initially I went slow and later I went fast. The biggest mistake I made was that people— I started with a slow machine and then because it was so slow I walked away from it. It took a while. I fell asleep. And I paid for it all through the next day. Don't do that. I would later learn that like paying a bit more for a faster machine, one, will actually make the whole thing train more than proportionally faster. I'll show a chart in a few minutes. But more importantly, it gets you to pay attention to it, and I found that to be really worthwhile. So this is the two lines of code. I sort of broke it up a bit here, and on the, on the slide I if you downloaded it. Clicking this will take you over to a gist on GitHub so you can play with this. I discovered there are two important things you want to do when you're fine-tuning a model. The first is you need to make sure you have the right packages installed. The most important ones I found are PEFT, Transformers, Torch, and of course AutoTrain Advance itself. So you may find for different models you're working with that you may tweak this a bit. But I found that to be, when I had those, it sort of stopped giving me grief. That was a one-liner for me. It takes like 5 minutes for those to install. Second thing I did was just run Auto Train. Once Auto Train is installed, this is the way you actually consume Auto Train. And there are more ways to use it. I recommend you go check out the documentation. But what I did here was just say, This is an LLM I'm training, right? I'm going to pull it from this, and this here is a reference to a Hugging Face repository. Right? So I'm going to pull the original data from Hugging Face. Small, medium, large, whatever it is. That's the first thing it's going to do is just download all those files. And then I just tell it where I want to install it into. And then this data path here is actually just a path to in this case, actually a CSV file, and it's looking for a CSV file that has a single member called text. It's customizable, but that's what it's usually looking for. So it requires you to go put together like the prompted question part, right? So it's like a level more complicated. But I actually found that really cool because I was using an instructor model like this. So what I did was I took some data from my corpus. My corpus, by the way, is on my office hours I was doing for my community, and I have transcripts from all those office hours. Now I'm just feeding in those transcripts and then using that to get trained up to answer questions the way I would. The— and then, you know, telling it, you know, a couple of key parameters here. I'm using path forcing, a method for doing fine-tuning. Using int4, this is actually a really key idea. Remember how I talked about storing a number that was taking 32 bits? There's something called quantization. Anybody here heard of the term quantization? Alright, if you have a 32-bit number, but all the numbers in here, and this is the thing to know about AI in general, they're all between -1 and 1. You're using 32 bits to describe numbers that are between -1 and 1. There's actually a lot of of dead numbers that aren't being used, dead values that aren't being used. And it's also a whole lot of precision when often it doesn't matter. Wait, is that super super tiny or merely super tiny? So the idea is instead of using 32 bits to describe that range of, you know, just 2 numbers of value, 2 integers of value, we use int4. How about just 16 values? The idea behind quantization is to say that instead of having all of that resolution where I'm going to use 3 2-bits and have, you know, whatever that number turns into, about 4 billion opportunities of either side. I'm going to have just 16 possible numbers. And the thing is, when you already have 7 billion different parameters, it still works pretty well. The loss that you get versus one that didn't have that is relatively small. It trains a lot faster and it actually interprets a lot faster because you can hold more of it in memory at a time. You know, batch size, that's just basically saying how much you want trained. And then the, you know, how much of a context window you want to retain. And then this merged after part takes the fine-tuned model, merges it back with the original model, and now you have something that you can just ship up to one of these places to be able to interpret. Your model. Two lines in Bash. And don't get me wrong, the first one takes about 5 minutes, the second one will take hours depending on how much data you throw in. If you throw in like 20 examples, it'll take maybe 5 minutes. If you throw in, you know, 100,000 examples, it can take maybe up to a day. But how long it will take will depend a little bit on what GPU you use. So this is actually an ad from NVIDIA, and behind it is actually the original model numbers. Like, this is the 2019 V100 and how the V100s got better over the course of following years. And this is the A100 that they introduced a year ago. Now, that's how much faster it is. This is how much more it costs on a per-hour basis. So don't cheap out. Would be my recommendation in terms of how you use these GPUs. Don't use them for a long time, but be willing to spend more per hour. By doing that, you will cut your training times by well more than what the marginal cost is of these models. And this is something I specifically say about running training, because training's going to take like hours, as opposed to running inference later, which you'll goof around with and figure out what generates you the best best returns. Do you switch afterwards when you switch? Oh, you mean after it's trained? Oh yeah, so after it's trained you're no longer going to use that particular Paperspace model anymore. I'll talk about that in just a second, bear with me. So AI21 recommends having at least 50 examples or else you're not going to get any interesting results out of anything you fine-tune. I recommend in general if you're going to go down the fine-tuning road you probably You probably want to have more than that. Now the question becomes, how do I get all that data? Well, it comes a couple different ways. The first is if you're like training on like a book, you can just cut up the book and feed it the book, a couple thousand characters at a time. This can work remarkably well. But then one of the weird things is you train it to do one thing and it kind of forgets to do others. So like that Mistral model I was talking about is trained on how to be able to answer questions well. It's called like an instructor model. And like I ask a question, it will try to give me an answer. It knows how to do that kind of structure. But when I fed it a whole bunch of data, it stopped being good at answering questions. It would try to just continue my statement, continue my question in the style of me, which was not what I needed, right? I needed to answer my question. So how do I get to answer my question? I need to not only give it information that is relevant to me, but I also need to have it, you know, keep practicing, if you will, how to answer a question. But that requires restructuring this data, right? And I didn't have the data in sort of easy, you know, question-answer format. So I used OpenAI. I said, hey, OpenAI, here is, uh, here's one of my sessions. Here's like, you know, 20,000 tokens of conversation with me and my, uh, and my members in terms of things they were asking. What were they asking? What were the answers? And it pulled that out. And I said, hey, can you format that into JSON for me? And it said, yeah, sure. And then it was very straightforward for me to turn that into what actually Mistral required for its format. So it's a little bit like I was spending some time training it, teaching it math, and some of it teaching it how to take a math test, right? I already knew how to take a test, and now I'm trying to keep it, make it better at the domain without losing that particular skill it had that it was built with previously. So, and then that becomes the kind of thing you're going to find yourself messing with as you start to fine-tune these things and start to make them more your own. I recommend having a budget. This thing, you can't do this for free. It just costs. I talked about having better GPUs. They cost, but they don't have to cost a lot. I bet you can put down like a $50 or $100 budget for yourself and not breach it and have learned lots of interesting things and create some really interesting models. It just doesn't need to be something that takes days and days and days. Remember, you're not making a foundation model. You're not Sam Altman or whoever you guys— whoever's trying to tell me now. You are, you know, you're just doing the fine-tuning part and that fine-tuning part goes much, much, much faster. One other thing about using good GPUs As you get better at this, I don't recommend starting with this, but as you get better at this, there are other services out there that can, you know, give you more of a marketplace for finding the best deal on GPUs. There's this one called Vast AI that I really like that I'm able to pretty routinely cut my cost for getting that same A100 by about half by making use of somebody's extra that's sitting in a data center somewhere. How much you're concerned about your data going over to a data center, that's entirely up to you. Some of them are all stateside, some of them are not stateside. You can decide what things make the most sense to you, but what I find is like by using the easier-to-use tools early and spending a bit more on that, I save more of my own time. Later, as I'm getting better at this, I can save money by taking my expertise and applying it to cheaper sources. And finally, I said it before and I will say it again, auto-shut templates. It's your friend. Set a timer for yourself just to back it up. But with all of these container-based services, they come with a— and it will automatically shut down if it hasn't seen interaction in like at least in the last hour or whatever. And you can set the maximum for how long you want to run. Make use of those. That will save you money and keep you under your budget. So you were asking about switching and what we do when we go from training to inference. And this is an example of code that we use for inference. Now I have generally found that like if you make something in Hugging Face, most of the time you're going to need to make a little bit of code. It's not a completely no-code experience to actually do the inference. But it doesn't need to be a lot, right? This is what, 5 lines of imports? 1, 2, 3, 4, 5, 6, 7 including the print for actually running the model. And this is actually a pretty good context to be answering the question, how much juice is required? So your question of like, hey, would you still use an A100? You could. It'll actually go pretty fast. But you might not nearly need it because it's a much simpler operation. And so testing that against— and actually even in the context of Paperspace, you can take your same notebook and say, well, let's apply it to this GPU and to this GPU. And then you're doing that for like minutes or seconds. It's going to cost you a very small amount of money. To be discovering the answer to your question of like which one's giving you, you know, best price for reasonable performance. But when we're running our model, running it in a notebook is often not all that productive, right? Doing a training process is like a one-off sitting in the model can be a good idea, but when we're thinking about how we're going to deploy, we want to have things that are a little more systematized. How are other people going to get access to it? There are some pretty cool services out there that allow you to do this in a serverless way. Which means, as opposed to say Paperspace, which really wants to rent you a machine, you can have things that are, you know, only going to charge you for the time the machine was running the function, right, that was doing the inference using your particular model. Modal and Replicate, I think, are two really interesting ones in this regard. Paperspace now has an offering called Deployments that also works this way. And the way they kind of all work is they're all making Docker containers, right? Paperspace Deployments is making Docker container and then associating it with an HTTP endpoint. Replicate has a framework called Cog, and this is actually what Cog looks like. This is, again, it's relatively simple, right? You have a file of imports and you can just set up what they call a predictor class and then it's just expecting that predictor exists in whatever the file is you've imported into cog. And it has basically two functions that matter: setup and predict. And the idea is that the very first time you spin up your Docker container, it runs setup. That could be like where you download like those huge weights files I was alluding to, or at least initialize. And then second is predict, and this would run every time. And any of you— has anyone here worked with like Lambdas or serverless before? Okay, if I say cold start, who here knows what I'm talking about? All right, this is the cold start problem. This has been run once to have spun up their Docker container, and then at that point, if you keep on asking more questions, it just runs predict. So the idea is that it can be slow to start if it's never been asked anything before, but then it gets very fast, or hopefully relatively fast depending on how much, what kind of GPU and how complex the model is, to be able to answer your questions after that. This is 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15. This is 15 lines of code to interpret a large language model that you made yourself. And that's kind of where I've gotten in my journey here. Uh, I totally appreciate, uh, you all, uh, sharing a bit of your afternoon with me. I'm glad to answer any questions you have. Um, and, um, and if you're interested in like the slides, we're happy to have— that's what the QR code is for. And I would love it if, as you continue in your AI journey, if you'd let me know how I'd love to know how you're doing by sharing either on Twitter or LinkedIn or however you choose to share. Thanks so much. Okay, any questions before we all lose service here? Sir? Have you found there to be like a good open source model for Suppose you wanted to give it your own private personal data and you just wanted to download a model onto like, I don't know, a couple terabyte hard drive and, you know, train it on that data and ask your own questions. Have you found a good model that could do that kind of thing? Yeah, so if you're going to do that, I would look for a 2.7 model. 2.7 is usually the smallest size for the large language models. I would look for— and LLaMA 2.7 is actually pretty good. And there's a flavor of LLaMA 2.7 that is actually already sliced. And the slicing is important because some of these models— one of the reasons why it's so expensive to train these models is because you have to load the whole thing to memory, right? Or big chunks of it to memory. And this one has already been pre-sliced so that it's designed to only run a little bit at a time. So it's a lot less memory sensitive. That way you could use like your, you know, the 1070 or, you know, whatever NVIDIA GPU you might have at home to be able to train up on this. You can try to do training with a CPU. I don't recommend it. If you have something that's like one of those NVIDIA GPUs, you can start to do some more work to put that all back together. You can even use AutoTrain for that job. You'd just be running it locally in your environment. And using that GPU using what's called CUDA, which is the library that connects it to the GPU. That would all be completely private because you'd be downloading the model, right, from Hugging Face, but the data would all stay on your machine. And then of course you can use like a notebook or a command line interface for the purpose of running your inference. So you can be asking all the questions you want. So yes, you absolutely can do that. You know, one of the other things I've seen lately that's really cool is starting to run on a phone. Right? It's starting to run on a phone. The trick is both in running on— phones are like running on Macs in which you don't usually have access to a GPU because it's a particular way NVIDIA set up the GPUs that allow these other libraries to work. Right? I'm sure eventually we'll get past it, but it's been a decade and we haven't yet. You know? So the— so all of which is by way of saying that if you're looking to do this, I recommend having a little bit of hardware. Right, having access to a GPU that your Python can work with, and that's probably an NVIDIA one. And I also, and then, you know, tinker with it and see how it's working for you. There's another really cool little utility called ollama that's out there that provides you with a better interface for a local app to be able to run a large language model in a local context as opposed to, you know, having to go down and write your own code for it. So that'd be everything I might check out if you're looking to do that really in a local private context. Sure. Other questions I can answer for the group? Sir? What do you think is the best way to, to get better at getting good results from like a ChatGPT or something? Like, is, you know, prompt engineering or course a good way, or just playing with it more and trying different things, or something else? I would do two. Okay, I'm a data person, so I always like to look underneath what inputs and outputs are, and I've actually found that to be a very productive way of looking at things like GPTs. You can now, even in ChatGPT, like right-click on a lot of outputs to find out like what was the actual prompt behind a prompt. And they kind of want you to do that because they want you to use more of the APIs, right? That's where they think they're going to get repeating this and that. So the otherwise it's $20 a month, who cares, right? They think that if you have that kind of interest, you're probably going to be a bigger customer for them. So I would check that out. Like, what is the beginning and the end? Juan Pablo from Microsoft gave a great talk earlier today where he was talking about what he calls meta prompts, which are actually the real prompts. You go type in whatever you're typing, what's happening is going to be injected into something bigger. Find out what the big boys do. A couple places you can go to find out where, what's getting used that I think are really neat. One is, I've alluded before to the AutoGen project by Microsoft. That is all based on prompt engineering. Behind the scenes is all OpenAI calls, and they use prompt engineering, which is, you know, MetaPrompt's wrapping up this data in order to construct the information, right, that's trying to pass for each of those transactions that's going to do into GPT-4. The cool thing about that is you can sort of skip through that, like, I don't understand this code, I don't understand this code, I don't understand this code, and then you see the prompts. It's just text. And you can see how those folks do it. The other people I would check out are LangChain. If you go open up the LangChain repository, you will find underneath a whole bunch of prompts that are created particularly for what it does. It does in like retrieval augmented generation, like when people were doing Pinecone initially, like about 6 months ago. And you will find that they, you know, people didn't know how to use it. So it includes both the code and the sample text for how to work with that. And then the third thing I would check out is something called AutoGPT. AutoGPT is a project that came out I think 4 or 5 months ago, trying to be the generalized solution problems using agents, like just try— you give it a simple problem and like it's supposed to figure out how to do it. I'm not going to say that it works that well, but what it does do inside the code is it shows you how it sets up its prompts. So you get to find out how some of the best people in the business have been setting up their prompts just by looking inside the source code. Source code that's much easier to read because it's just text as opposed to being broken up into a whole lot of classes or functions or whatever. You don't need to go all way down that rabbit hole. That help? Yeah. Cool. Everyone, thank you so much. I really appreciate this.
