Prompt-Engineering for Open-Source LLMs

Sam hi everyone, my name is Diana.

Chan Morgan and I run all things community here@deeplearning AI. Today we have a very special speaker to talk about prompt engineering for open source LLMs. As you all know, your prompts need to be engineered when switching across any LLM, even when OpenAI changes versions behind the scenes and this is why people get confused why their prompts don't work anymore. Transparency of the entire prompt is critical to effectively squeezing out performance from the model. Most frameworks struggle with this as they try to abstract everything away or obscure the prompt to seem like they're managing something behind the scenes. But prompt engineering is not software engineering, so the workflow is entirely different to succeed. So today in this workshop we'll learn a lot about prompt engineering versus software engineering, open versus closed LLMs push accuracy by taking advantage of prompt transparency, Best practices for prompt engineering, OpenLMS and RAG and how to implement RAG on millions of documents. This workshop will be recorded and the slides will be sent out afterwards. It is inspired by one of our courses that we have made with a speaker for fine tuning LLMs. Our event partner today is Lamini. Laminai is the all in One Open LLM stock fully owned by you. At Laminai, they're inventing ways for you to customize intelligence that you can own. And now I'd love to introduce our speaker, Dr. Sharon Jo. Sharon is the co founder and CEO of Lamini. As a former Stanford faculty member, she led a research group and published award winning papers in generative AI. Sharon teaches some of the most popular courses in Coursera including fine tuning LLMs and has reached a total of over a quarter million professionals. She received her PhD in AI from Stanford, advised by Dr. Andrew Ng and before her PhD was an ML Product Manager at Google. She received her Bachelor's from Harvard in Computer Science and the classics. Finally, Sharon has served as an AI advisor in D.C. and been featured in MIT technology reviews throughout 35 under 35 list. We're so happy to have you here today Sharon. Are you excited to dive into everything?

So happy to be here. Thanks for having me again.

Of course. Excited to have your little llamas, you know, stick out and help everyone out.

Everyone learn a thing or two from them. They may have been making cameos throughout certain deep learning short courses, so stay tuned.

Amazing. Well, why don't we get started? I'm sure everyone is super curious to learn everything that you have to teach today. And for anyone that has questions, we drop a slide in the chat where you'll Be able to vote upon which questions you want to hear from Sharon the most. All right, take it away, Sharon.

Fantastic. Yes. And please ask questions. Yeah, very, very open to that. And especially if it's a little bit spicy, I kind of want to see some of those questions. Cool. So I'm Sharon. I'm the co founder and CEO of Laminai and I'll be talking about prompt engineering for open source LLMs today. As the title suggests, prompt engineering is not the same for closed versus open LLMs. And I'll go into exactly what that looks like. And this talk was really motivated. I know Diana gave a really good intro of me, but this talk is really motivated by the pain that I had both seen and felt myself. And I just don't want other people to go through the same struggles. So this is to also step through some things that I've learned throughout this journey. And also I've seen, seen a lot of people, a lot of smart people kind of mess up and, and spend, you know, too much time on like too many months on. So I believe in bringing people on the right path, path to success. Of course this is, you know, this is, these are my learnings. Of course there are a few, few hot takes in there and you'll see them very soon. So this, this should be a fun talk. Okay. Yeah, my background, I think that was basically covered, but I was also the first generative AI instructor at Deep Learning. Um, so I love, I love generative models. I find them really, really magical. I also love people. And so my mission in life is actually just to make it possible for more people to be able to wield this magic that I, that I basically fell in love with during my PhD. Cool. So just to go over a little bit, you know, hierarchy agenda of what we're going to want, what we're going to cover. So first is, you know, how, how is an LLM, you know, how it was fine tuned or what was going on behind the scenes? Behind the both API, but also before you started touching that LLM, what was going on behind before that, that has now impacted the way you prompt it. So what goes on behind the scenes, it does impact your prompt. And I'll go over exactly what that looks like. And you know, there are different LLMs and they actually have different prompts that do better on them. I see people saving prompts for GPT4, for example, and thinking that'll generalize to other things, other models, and that's not the case. And actually even different model versions. Right. Like people have Felt that deep pain around that. People think it's, you know, oh, where is this backwards compatibility which is a normal software paradigm? But you just don't see that here with large language models. And I'll go over that a little bit. Something I want to hit home also is that prompting is not software engineering. And I kind of see people really struggle with this because it's called prompt engineering. I think that was poorly named from my perspective. I think it's closer to actually running a Google search query where you're iterating over that query. And I'll just talk through exactly. Best practices kind of what I've seen. Finally, prompts are just strings. I really, really do not want you to overcomplicate this. I'm going to be super spicy about this and slightly snarky about this during this talk, but prompts are just strings. So if there's one thing that you take home is that prompts are just strings and you can handle a string, I guarantee it. Okay. And finally, rag is a form of prompt engineering. A lot of people get that confused. They're like prompt engineering versus rag versus fine tuning. I'm like, no, this is a form of prompt engineering. I'll tell you why. It's because it just impacts the prompt. You're concatenating strings to a string. So again, do not overcomplicate so much. And I see people over complicating way too much. So this is kind of the agenda we'll go through. Who is this for? I mean, I posted this a bit on Twitter. This is largely for awesome, really smart software engineers. I hope a lot of you there are this archetype. You're extremely good at software engineering and you're curious about working with open LLMs and it's very exciting time to do that. And there are many, many benefits to working with these models that I can get into. But I'll just touch on a few that are relevant to prompt engineering in this course. And who's this not for? Well, this is actually not for people basically like me, who's trained thousands of generative models and proposed new architectures because and train them successfully. And that's because that process helps you understand why the prompt acts the way it is and why you have the prompt the way you do. And I'm kind of, my view is you shouldn't have to go get a Ph.D. do all this stuff do to understand this. This should be, this should be made more simple. And I'm going to try to go through how simple it is through this class. So it, you know, I just want this to be simple. I don't want you to go spend so much time on a PhD. I want you to be able to listen to me at 2x and get it done in 30 minutes and be entertained. Okay? So let's start with a memorable analogy. This is really starting off really spicy. So I was thinking about this last week. I was like, how am I going to communicate this? You know, what is the best analogy? And I apologize. This is my best right before I fell asleep. So you go out every day, you're wearing pants, okay? Everyone sees you wearing pants. You see everyone wearing pants. You're told it's right to wear pants. Like it's the correct decision to wear pants. You are a good person for wearing pants, okay? Like you're just a good person. Every day you, you act normal because you wear pants and you see other people wear pants. You told this right to wear pants. It's told is correct, moral even. But then one day you're not wearing pants, okay? You go out, you're not wearing pants. My question to you that I'm going to pose right now is do you still act normal? Right? Do you still act normal? Okay. And this is like, maybe like an extreme analogy, but this analogy is LLMs have the same wear pants feel. So like elements also wear pants. Okay? And that's via their prompt setting. And I'll go into exactly what that is and remember that a prompt really just is a string, right? It's just a string. So a string, you know, a lot actually matters for a string. Sometimes it matters more for the LLM based on how it was fine tuned or trained than it does for us. But even for us, I think even small, you know, edit distances within a string can make a big difference. For example, here's Yann Lecun's recent tweet of Open Space AI versus OpenAI. I know, it's very spicy. Just making this very spicy for you. So, so you know, very different words, different words, different strings, only by a little bit of edit distance. So wearing pants is an analogy for prompt setting. And every LLM, every version of an LLM has a different prompt setting. Maybe not like entirely different, but it's been adjust trusted to learn a certain, certain notion of wearing pants. And okay, let's just go through a real example. All right? Mistral, one of the great Oban models today. Very exciting model. Okay, so let's say I want this prompt, I want to put this prompt in, respond kindly to the Child, I really hate zucchini. Why should I eat it? Okay, to be fair though, I really hate zucchini. Why should I eat? I actually took that from my co founder who's awesome, but an adult. But I just say respond to. Respond kindly to this child. So like, what do you expect? You expect it to respond kindly to the child? Right. Okay, well let's see. Without pants, what's going on? So without pants it just like goes on in the third person. First, it's important to remember that everyone has food preferences and it's normal for children to not like certain vegetables. That's just not good. It's slightly off, right? And we'll look at examples where it's way off. But this is just a simple example where it's definitely off. And just if you were to just let the LLMs or you put the pants on the LLMs, it actually just responds to the child. And so I understand that you don't like zucchini and it's okay to have food preferences. However, it's important to remember that eating a variety of good foods. A variety of foods is essential for good health. Okay. It's not perfect, right? But it is better and there is a shift immediately. And as you'll see soon, putting on pants is super easy. Just like it is for you do. It's easy to do for the LLM as well. Again, strings. Okay. Same thing goes for Llama. You're a health food nut. I'm drinking green juice. So you're just like putting this into chat GPT. What do you expect? You expect it to be like, oh, you know, like giving you advice, etc, without pants. Unfortunately, Llama two here just like continues the sentence, right? And from my perspective, I'm like, oh shoot, you know, it's not instruction tune. Maybe, you know, I'm a little bit, you know, judgmental that it's like I'm also feeling a little judgy. Okay, okay. It's just not good. This is like kind of gives like this off the rails feeling. And I know people are using LMS at scale and so just this difference where now it's able to respond. I know, I know. Llama 2 is very, very expressive. It's kind of hilarious, but it's very expressive. But this difference makes a huge difference at scale as well. Right? Like when you're using these alarms. So let me just share with you exactly what pants mean. Okay, so every has a different notion of wearing pants. You'll see soon that you, you can actually have control over what the pants look like. But you know, at a very, at a very basic level, every single LLM has a different, different notion here. And for Mistral, it kind of looks like this. For llama 2, it looks like that. And what is, what is exactly happening here? Well, you know, these are, these are called meta tags. So if you go to a party with OpenAI people and anthropic people, you might hear them talking about meta tags. And what a meta tag is, is like here. These are tags inside of your prompt that indicate something about what the user put in. For open LLMs, this is completely exposed to you or transparent to you or available to you rather, unless you use the framework. So I'll go into that later. But this is possible for you to just see and use and. Yeah, let's actually, you know, you know what I want to do? I want to dive right into it. So I think they can drop a GitHub link to you. I'm going to share my screen and just show you some code and just, just walk through exactly what, what do these pants look like? Okay, so I'm going to share different screen. Okay. Okay. I believe I share the wrong screen. Unless I didn't. No, I share the right screen. I apologize. Cool. Okay, great. This is an open source library. Super basic stuff. It's like, oh man, just 50 lines of code. 60 really simple stuff. All it's doing is I am running a couple open models. First I'm going to run Mistral with you. And then I have a few prompts here, kind of just like the green juice one that we talked about, the zucchini one we talked about maybe a couple others. And these are, these are actually real ones I just grabbed from my own history. So like I did not spend that much time on this. But it's pretty apparent it's pretty slight difference between using the putting pants on or not. And so all I'm doing is iterating through printing what the prompt is, letting the model generate based on that prompt. And then I'm adding pants on pants on here, printing out the pants with it and letting the model run it with pants on and then comparing them. Same thing with llama 2 here. Doing the same thing here. There's something fun with the system prompt and the user prompt. Cool. And you'll see here I do, you know, some kind of. This is my own personal convention. And again, these are just strings. Like all you need are F strings. And these are just like 30 max, 30 extra characters that you need. So I just don't understand why people need like extra frameworks around it. Okay, so here, let's. Let's focus on this guy. And I just ran it up here, but I actually am just. Let's just run a live. Why not? So let's do this. Let's import. This is a class that runs different models. And so I'm just going to run this draw here and which one? So I know we kind of looked at those other two, so I'm actually going to do the code ones because I actually thought those were really interesting since they made a massive difference. And I know that people are kind of working with code. So if you were to do this here. Let's see. Let's run the model with just. Okay. Oops. Let me just say prompt equals this. And then I'm going to run the model. So this is how you run the model. Element generate and just that. And no pants on. Okay, great. Then I don't like looking at new line symbols. I'm just going to run print. All right, so totally off the rails. Great. So VS Code. This is something again. I wrote into an LLM previously how to find code that's been checked in a long time ago, more than six months ago. I was trying to figure that out, but then here is just a disaster. What is going on? I have a large code base. It just starts to think it's me. And then this get log stuff, it's like, what is happening? Right? All right, so let's put some pants on this guy. Let's see. Okay. And actually just one note here. I find it really interesting that people don't know this as well, but like really interesting that you know, just a single space here, this matters a lot. And just like a fun historical Note back when GPT3 was out, but no ChatGPT. So like this was circa. This was circa 2020. And I was playing with these models. It was so interesting because like you could put. You would end your whole thing with a. With a space. Like imagine like if you ended your question with to chatgpt with a space, it would go completely off the rails. It'd be no pants on already. So like very sensitive now. Models are way less sensitive, including open models. So it's just, it's kind of exciting to see it. But before the pants thing used to be more extreme. Let's put it that way. Okay, so let's, let's actually do this. But yeah, so I, I just found that like historical note really interesting. We can even run it with and without a certain space here. Like if you want to run it, okay, not here, but like if you want to change what the pants look like. Sorry, I need a friend. If you want to change what the pants look like and you remove a space, it should give you actually a slightly different string. So just know that these things are sensitive to the strings. It's less sensitive Semant if you were to embed everything, but it's still going to give you something a little different. Happy to go into the machine learning reasons why. But just so you know, it is different here. Much better. Wow. In visual studio code you can use the built in git integration. You can do all these steps. It's possible. This is actually not totally true, but this is a valid response. I would be like, yes, I'm talking to someone who is wearing pants. Is kind of my thinking. Great. Okay, so you can run a bunch of these also on your own. This other one, this example JSON file, I also was running this on my own. This is just kind of. I, I literally. This is like literal code I have, I've written and I needed help with and it was a dramatic difference and maybe I'll just scroll up to go look at it for you. So like put the script just so you can see what the script actually outputs. So it would be something like, let's see. So like here prompt, you know this is adding pants on. Then it's actually like producing code, like legitimate code versus when you don't. It doesn't do that. Just run it like that for you to see. But yeah, I just find it just interesting that I think people miss this and then they think these open models don't work and that, that's actually really unfortunate because they do work. They actually work very well. And you can get like a, just a massive performance bump just from knowing this. So this is what the exact script does in the open source repo. So you can just say you can see the prompt. Then you can see the prompt with pants on. You can see the response from those. And yeah, you can see like, like why are you saying a colon? Like stop. I actually know why, but it's fine. Let's see. So yeah, this is off the rails. This is better. Let's see. Great. Okay, cool, cool. Here I'm going to go back to the slides. I think the point was made. But feel free to play with your own. There are definitely going to be things that are a little bit more sensitive. Things that are going to be less sensitive. But just something to note if you're, if you're using LLMs that way. Great, Cool. So the learning is if you didn't get it, the 10th time is elements want to wear pants. Okay. So like give them the pants to wear. And you might be wondering, okay, well why didn't I know this? Why have I been basically abusing my LLM and not letting it wear pants? Well, it's actually pretty poorly documented. It's not really in a lot of official releases. It took actually our team a bit of time to find as well. It's just not. It's also when it's not official, you're like, is it, you know, is that the right one? Is the one that like. Because AWS was like a partner with meta swam release, like is their blog right. So, you know, a lot of questions there. I've also worked with folks who, you know, they, this is terrible. But they were like, oh, our model does so much better than the base model. And I was like, well, I got better base model results than you and it's because they got the pants wrong. And I was like, so your model is not actually better than the base model. So that's. It's important to know that this is not something you need for close alums. I will go over, I'll go over that with you in a bit. You can argue it's implicit. Lock in. Right. So they like manage it all for you. I think it's also so that they can push out new versions and I know that people don't like it when they push out new versions and update things behind the scenes because it's not transparent. At the same time, the argument is if you were to create your own, you probably would do the same because the software expectation is you don't want to pip, install, upgrade every five minutes when these models are constantly learning. There is a trade off here, to be fair. And you also thought your framework could handle it correctly and a lot of frameworks don't. And I've been, I've been guilty of doing that. So just getting the string representation right isn't always the easiest, but it's, you know, making it transparent, I think can help you get just a few extra characters in 30 extra characters. So yeah, onto why it's different from closed models. So ChatGPT, you can text ChatGPT, you don't need to add pants to it. You know, it just it chooses to wear pants behind the scenes or OpenAI is putting pants on behind the scenes. So this is actually what the API looks like. You are putting in a. I would say Dictionary here, but mainly this string of content. What's happening behind the scenes? This is my guess, this is not actually it. I mean, if this were, that'd be kind of weird. But so like, maybe there's a role thing going on here, maybe there's a message tag there, right. But there's some representation within the string that they're doing behind the scenes. And you know, if I were them, I would probably make those representations, single tokens that they've already, they've figured out kind of like an endless sentence token so that you can like represent it very minimally in the prompt to be token efficient. And so, so this is a guess, I don't know what exactly what it is. Right, but this is kind of what's going on behind the scenes. And this is why you don't think about it when you're working with closed LLMs, right? So you just, you don't think about it. And if you're shipping your own close LLM, like this is also something for you to think about. Like this is a trade off, right, for your user. On the one hand it is kind of gnarly to look at those tags for like a non engineer, non software engineer and you can't wrap it in a beautiful interface. But on the other hand, it does help with working with them and I think there's real benefits to understanding that those exist and that and what they're used for, right? Why, why you need them. So something I just want to hit home is at the end of the day though, all you're adding to the string, you're like, oh, wow, great. ChatGPT handles it for me. Well, you know, like on the one hand, yes, it handles it for you, but on the other it's like it's just a string. It's like 30 extra characters. You can do it. I believe in you. I believe in your ability to be able to concatenate strings. And you know, I was trying to look for, you know, what level is string? Like working with strings. It's actually a kindergarten level. This is from scratch. So like no excuses, you're a software engineer, no excuses for not being able to work with strings. Okay, great. So let's see. 80% of frameworks overcomplicate the string though. So I said, do not overcomplicate string. 80% of frameworks get it wrong. It might be a few characters off, it might be completely buried somewhere. So this is, this is LinkedIn. I know I'm calling them out, but I'm actually guilty of it. Too. So I shouldn't just be calling out other people. I've been guilty of this too. Our team kind of built a kind of type system to make it easy to work with LLMs where we like converted your type to input output types. That was very pedantic. Using pedantic is very pythonic. And you know, you know a lot of software engineers like, wow, awesome. I get to work with LLMs this way. But to be honest, it was just totally overcomplicated and we spent like months ripping that out over time slowly. And so I just, I don't recommend that. And so that's my learning from the pain that I've seen. And I think there's like, I don't think it's potential. I think it's to, you know, help with working under normal software paradigms. And I think that's, I think that's valid. I think there are better and other ways to do that where the prompt doesn't get totally buried because at the end of the day, you're going to maximize performance when that prompt is transparent and you're able to verify as the person running it that, that, you know, all the characters are in the right places. And if we were to go a little bit more advanced, I would tell you ways and ways how to veer off of those pants and maybe, you know, change the color of the pants to follow the analogy. But like this or where, where skirt. But yeah. So it'. I think it's important to keep the prompts transparent, especially for open LLMs. Cool. So the learnings keep the problem transparent. Wow, I'm so ahead of myself. Make them easy to change because every new LLM, you're going to change them. That's for every new version, every new update. I don't know if this is the case now before a very long time on hugging face. New versions of models, and this was just last summer, new versions of models would come up without people saying anything, without the creator saying, like, oh, we pushed a new model to the same name, so it's going to be the same one. There's no backwards compatibility on that when someone else is owning the model. Just note that that does happen. It's the same thing as OpenAI, changing versions without changing the name. Just know that prompts are, are the highly variable part of your workflow. So I want to make sure that you understand why this is important. Like, because it's so highly variable, it needs to be transparent and easy for you to access, to be able to iterate on it constantly and it's not a pain, it's a string. So like, if it's, it's not a pain in the butt to do this, it is a string, you can do it. And it actually gives you a lot more flexibility in terms of working with these models. Okay, so, so we've gone over a few different things. I, I feel like probably what's on your mind maybe is who decided how the LLM should wear pants? Like maybe you saw. Okay, well, clearly OpenAI decided some things for their models and maybe Meta and Astral decided some things for those models. But who gets to decide? And I'm extremely, I extremely opinionated on this. I think, I think everyone should get to decide at some point. And I think it depends on your application and you know, it depends on like what level you want to be able to make that decision at, right? Like maybe maybe you do want to make it at the same meta level, but maybe maybe you want to make it at a higher level, right? Like another, you want to add another prompt setting. You want to add another notion of wearing pants for this LLM. And the way to, to do that, like the, the standard way of doing this is fine tuning LLMs. I also do teach that class and I plan to teach a class that goes into that more in depth for software engineers. But essentially that's how you can decide the prompt settings because it shifts the knowledge and behavior and you choose what this meta tags are. And who decides? This is why enclosed LLMs. That's why you feel like your prompts are breaking or not backwards compatible. It's because someone else is deciding how to wear the pants and things just shift a little bit. For every update, there are actually infinite prompt settings you could target infinite and they're all just represented in strings. So it's actually not that hard. So we looked over chat, so those are chat templates, essentially I call them prompt settings because the word, the phrase prompt template is just so overloaded and used for other things as well. Like not for this. I've seen chat template. I've seen, I think Mistral released something with or hugging face together with hugging face. They released something to help with managing it. And I do think when it's very simple and you can verify it, it's okay. But you can also produce JSON outputs. You can function, call, it's all fine. These are all prompt settings you yourself could target and control. Oh, I added this one because I thought it was fun because one of our customers was kind of fine tuning a model towards menu items. Because they're extracting menu items from a conversation that they're having. If you imagine like a drive through window kind of thing. So it's kind of fun. So you can do anything. Like what I'm trying to say is like you can do anything not just like the standard stuff, right? So like whatever format you want. So you can use fine tuning. And what that does is have your model view another setting as wearing pants. Or you can have it forget a previous setting. You can do whatever you want basically. So kind of a powerful tool to do that. I kind of want to add this like little tidbit of something a little bit more advanced for those who are watching, who are a little bit more advanced on this. You can actually get a model to think it's wearing pants without fine tuning. Called this Emperor's New Clothes, if you know about the myth or whatever it's called the story. And we actually built this, our team built this. And, and we have this custom inference engine that is able to take any model and without any fine tuning be able to actually produce guaranteed JSON output. And so you can actually basically force some pants on. It doesn't fit really well or you kind of trick it into it, but you can get guaranteed JSON output. So this means when I say guaranteed, I mean like 100%. Like, it is not like it's not even probabilistic at that point. So yeah. Cool. So the learning here is that, you know, maybe you're like, I just don't want to deal with this and lazy use. Like I just need someone else to manage 30 extra characters or like 10 or something. Okay, whatever. I'm going to stop ranting about that. We're curious. Use. Like, I actually want to control this. I actually want to understand this better. I actually want to push performance. It's okay to be lazy. I can be lazy too. For different use cases. You want to be lazy. Like if I'm just like randomly asking a question. In other use cases where you're building your own maybe for your company or your side project, etc. You do care, right? So it depends and usually I transition between. I start lazy for everything I do and then I moved curious. So, okay, so prompting one strategy for prompting that I just want to make sure we hit home here is around iteration. So for software engineering, you know, how do you be a good software engineer? Okay, you create a good design and you implement it. Okay. And a lot of your time is in creating that good design, right? And then implementation, like some people are like, oh, get up, copilot. Not going to replace me because most of my, most of my job is just designing, architecting, right? The code. The system before I write the code for it. Great. And 80% of devs I meet are actually prompting like their software engineering. So I actually find that, you know, some random person off the street might, might be, I'm not saying actually is, but might be better at prompt engineering. And I find that just so interesting because for prompting, there's no such thing as designing the right thing. You can't take a class and be like, can't take this class ready. You can't take a class or this class or any class and say, like, I'm going to do exactly these exact things that they said. Because at the end of the day, it's a very, very iterative process. You're not designing it right, being perfect and then implementing it. You actually should just give something bad. You should actually be lazy. You, you should be lazy. You. So like, you should give something bad. Just like when you're googling, you know how you don't even spell things right sometimes you just be lazy and like iterate. Oh, oh, Google didn't understand what I said. It didn't do the. Did you mean this? It's like totally didn't understand the random keys I pressed. But great, so let me like, iterate on that. Make it better. Make it better. And so that's the mentality. You have to go in to the whole, whole, whole thing of prompting. And because it changes so much across different models, versions, etc. You want to be able to have control of that, right? You want to have transparency over that and you want that to be easy to access. So there's constant prompting you exactly what I said, a good enough strategy. So I see a lot of people plateauing and having trouble realizing that the best strategy might just be to time box it and to realize you get diminishing returns. And I think the way you do that is prompting iteratively to learn that naturally you'll just do this. So figure out the time. It's probably one hour. Figure out the time though, and invest that time into it and then get diminishing returns after, you know, 100 iterations in that one hour. Okay, so just like you're googling, like really not anything different. Don't spend so much time just designing it or like watching all these videos to design it. And again, because the prompt settings can change, like things can change. It's actually, it's good to learn this skill and it's good to make this very easy on yourself. And so yeah, I just think it's make it, make it super easy in your code base when you're working with models to change these prompts because you can't leave it stale. Unless of course you're controlling the models themselves, you're fine tuning themselves, you're checkpointing themselves. And that's actually one of the big benefits of open models, right? So you can control when the versions upgrade. So the learning is. Prompt engineering is neither software engineering nor is it machine learning. I can also guarantee you it's not also not machine learning, despite all the papers these days of like people publishing prompts, just not. Yes, there are some fun things that you can learn about the prompt where you can think step by step, for example, but it's not machine learning. So this is all the stuff you put into the string before, before you actually run the machine learning part, before you actually run the model. So again, just string manipulation, kindergarten, scratch. Cool. Let's, let's move to rag. This is also something I find people trip up on. They think it's a completely different thing from prompting, but I think it actually is a subset of prompt engineering. It's a way to prompt things. So RAG is a search, it's information retrieval. It's been around for decades, actually longer than AI, way longer. And speaking of like prompting, like Googling, Google is the best at this, at search. And so there are a lot of good practices, a lot of best practices that you can employ from decades of research and also decades of real production deployments. It is at a very basic level. And I want to again hit this home because I think people get a little bit intimidated sometimes or they think it can do all these magical machine learning things when it really can't. It's just concatenating more strings to your string so you know, don't overcomplicate it. And also at the end of the day, it actually is good to have transparency around this. I see people running big workloads without any transparency, without seeing their prompt at all, when you know, they bragged it and added relevant content to it. And they could have been debugging there at that level. They could have caught so many bugs. Right. So, or ways of just representing the data differently, adding, adding a title, adding a header, you know, some, some other meta information within every single chunk. So don't overcomplicate it. Like you can do it. You don't need a PhD machine machine learning for sure, because it's not Even machine learning. So great. So I put together a really simple repo for us to go over and kind of the goal of this, I say 80, I think it might actually be 70 now. Kind of proud of that, but. So the goal of this is to make it look simple, but also actually just see how simple it is and yeah, kind of go through what would a. This is. I mean, obviously kind of like what a machine learning person, how a machine learning person would go about it. Because I'm using Feist, but essentially how would I go about doing rag? So I'm using fais. This is an incredible library by Facebook or Meta and it's been out for a really long time. I should have checked exactly how long, but way before ChatGPT. So it's for similarity search. It's kind of their embeddings library. To be able to handle embeddings well, I'm importing some standard libraries. I'm also importing our embedding API to be able to hit embeddings really quickly and also being able to run a model on the outputs of something in rag. So like after your whole prompt has been basically compiled or created with rag, be able to run the model there. Okay, so exactly what's going on here? So I'm just going to set your code so that it all looks transparent. And again, 72 lines and this part is just running it. So cool. Cool. So loader. This is just loading data. I'll show you what that looks like. Build index. This is an important function to understand. So really it's grabbing different pieces of content inside of your data. So maybe you have. Maybe you have a. TXT file with data in it and you're loading it up. Let's say I know there's another file. Maybe it's cheating for the 80 lines, but all it's doing is chunking and loading, loading files. I'm putting the chunks in batches just to be more efficient. And then chunking the files and chunking each file. Cool. And then you iterate through that. These are batches, right. And what you want to do is just like get embeddings of them and you get embeddings of them. So these are vector representations, AKA just like numbers representing the semantic understanding of every chunk. Is it perfect? No. Right? It's not perfect. It depends on what you give it. Maybe you can give it extra information so it can represent more. I know there are a lot of techniques about. You know what best way to do it. There are a lot of different indices you can use, and you're welcome to go look into that. They often are trading off things like, this is a really simple one for our purposes, but they're often trading off things like, you know, memory or efficiency versus accuracy. So a lot of different techniques there that have been around for decades and that people are using today. So. And I think, like, the best ones are not being invented today. The best ones were already invented a decade ago. So. Yeah, so. Or most, you know, most popular ones, certainly. So I'm putting that into that index, adding it there, and that's all I'm doing. And then here, all I'm doing is extending this list to include the actual content so that I can retrieve it easily. That's it. So I get the embedding. It's basically a representation between the embedding here, you know, semantic understanding of. Of. Of your chunk of data and then being able to actually get the data associated with it. Okay, what's going on in embeddings? All it's doing is similar to calling a model so you can do that. This is probably going to be your biggest bottleneck in rag. It's not going to be building this index. I don't think these models are getting larger and larger. Of course, you can get away with running smaller models. Right. So I think, I think throughput here matters. And so here you get.