Prompt Engineering Workshop

All right. Uh, by way of introductions, that's me. Uh, I've had several different careers at this point— aerospace, search technology. I wrote a book for, uh, search at one point and swore never to do again. Uh, at GitHub, did code search. I worked in data science with Hamel Husain. Uh, then got to work with Copilot. And I'm writing another book, which I swore never to do again. And now I've just left Copilot. I'm going to wrap up my book. And I'm joining the ranks of Hamel Husain and Dan Levenson in trying to see if I can be a good consultant for LLM applications. So that's me. Now, what's a large language model? Who better to ask than ChatGPT itself. So I asked ChatGPT, and it said a large language model is a type of artificial intelligence system that is trained to understand and generate human-like text. It learns structure, grammar, and semantics of language by processing vast amounts of textual data. The primary goal of a language model is to predict the probability of the next word. You know what that is, right? That's that goofy button on the middle of your cell phone when you're typing a message that predicts one word ahead. It's a really simple idea. So how on earth is this idea taking the world by storm right now? Well, some of that is hiding in this word large. It's not just a language model, it's a large language model. So this is compressed, and I'll, I'll just have to go through it pretty quickly, but a lot has happened in the last 10 years or so. It— around 2014, I the state of the art was recurrent neural networks. They had an encoder and a decoder, but there was a problem. There was a bottleneck that made it difficult for the decoder to look back at whatever it wanted to look at. All the state was hidden in this vector effectively between the encoder and the decoder. So later that year, someone created this idea for attention. It was a useful way of looking at the piece that is most relevant rather than just packing everything into the space between the encoder and the decoder. Then Google said, well, that attention stuff, that's great, let's, uh, let's just get rid of all the other stuff and say attention is all you need. And thus was born the Transformer architecture. So, uh, as we moved on, a lot of neat stuff came out of that. The The encoder side of that became BERT, and that was very useful and continues to be very useful, including the new trends in search and RAG and stuff like that. But in June of 2018, they figured out that you could chop off the right half of this transformer model, and you have what we have come to know as GPT, generative pre-trained model. Now, the name is interesting. At this point, generative pre-training is almost a misnomer. But what it meant at the time is it's a generative model based on just training it on whatever text data you have. Typically, you would do a train a model, and then you would fine-tune a model to a very specific task. The training, it doesn't have to be labeled, but the fine-tuning is much fewer labeled items, and that's the thing that really makes the model good. Well, we started to notice something really unusual about these GPT models by the time we got GPT-2. In this paper by OpenAI, they introduced GPT-2 with a very unusual line in the blog post that introduced it. The very top of this blog post, it said, our model called ChatGPT-2, a successor GPT, was trained simply to predict the next word in 40 gigabytes of internet text. Due to concerns about malicious applications of the technology, we are not releasing the trained model. How on earth do you get from a model, you know, just predicting one word ahead, the same thing as the middle button in my phone, to this horrible concern about the future of our world, existential dread. Well, if you look a little bit deeper, it turns out that these things were— even though they were pre-trained, they were not fine-tuned to a specific cause— they started beating the state-of-the-art for the very specific trained models. Missing word prediction, pronoun understanding, parts of speech, speech tagging. Text compression, and then obviously, I mean, here we are in 2024, you can do summarization, sentiment analysis, all sorts of things, even though it hasn't been trained yet. But with great power comes great responsibility. Because these models are so crafty at all these different tasks, they can be misused. And do all sorts of horrible things as well. So that's why they put this scary line here to warn us that these things should be handled very carefully. All right, so with introductions and the big picture out of the way, stuff that you guys probably already knew, let's get into kind of the meat of this talk. In the next few slides, I'll go over several different techniques for prompt crafting. And then as we get about halfway through the talk, we'll move into some of the more recent things. Everything is moving towards ChatGPT, and there, there is a, a very important introduction of function calling in the middle of last year. So we'll talk about that and talk about how all of this can be used to build large language model applications. It all starts with the prompt. All right, so prompt crafting, technique 1. The first way that researchers started to realize that you could influence and control these things is by few-shot prompting. Remember, these things are sophisticated statistical models that are predicting the next word in a document. Now, if your document happens to have a very predictable pattern in it, then you can actually, by controlling the pattern that's set up there, you can actually control the output. So if you wanted a translation application, then what they would do is they would put in several handcrafted examples of translation, English sentence, Spanish sentence, English, Spanish. And then the actual task would be tacked on to the end of the prompt. In this case, I want to know how to translate, can I have fries with that? Puedo tener papas fritas con eso. But it set up a pattern so that the logical conclusion of this pattern, one word prediction, one token prediction at a time, is achieving the task that you wanted. Side note here, guys. I think you guys all have access to these slides. I put copious links to everything. Every, every one of these slides has several links hidden in it, so make sure you grab the slides. There's a lot of good reading material too. All right, so that's few-shot prompting. The next big thing is chain of thought raising. One of the early things that everyone noticed about these models is that even though they're really good at predicting plausible next words, They weren't terribly good at reasoning, at just normal logic. They were especially bad at math. And so an example of this are these little, little goofy word problems. For example, if we say it takes 1 baker an hour to make a cake, how long does it take 3 bakers to make 3 cakes? Well, a statistically plausible quick answer to that is just 3. That's if you're not thinking, you might even say that yourself. But it's the wrong answer. It's still going to take an hour to bake all those cakes. So with chain-of-thought reasoning, they use few-shot prompting again, and they built up several examples of giving the model a similar question. And instead of, you know, having the model just say an answer, they would put in the voice of the model, they would say, all right, here is how you think think through the problem. And this is just, you know, for lack of space, I've only put one example. But in this case, we have, you know, Jim is twice as old as Steve, blah, blah, blah. And the answer, rather than just saying the answer, Steve is 6, we have the model actually think through the problem and we write it out as kind of a verbal algebra problem. But this sets up the pattern. Again, the model is predicting the next word. And since there is a pattern in this document for thinking slowly and deliberately to the answers, then you're much more likely to get a long-form answer like this. And, and it's kind of interesting what's actually happening here. Once you peel back the COVID a little bit, these, these models don't have any internal monologue like us. If we were given this problem, then we don't just jump to an answer. We reason about it. We visualize the problem in our head. We talk to ourselves. You know, talk through the steps of this problem. We don't say it out loud. But since these models don't have any sort of background reasoning, they just, you know, every single token is the same calculation, chunk, chunk, chunk, chunk, chunk. Then by encouraging the model to, and by conditioning the prompt to spit out an elaboration of the concept, an explanation of it, then effectively what you're doing is replacing that internal monologue of the model. And it helps the model to have more of a scratch space and come to a more reasonable answer. In this case, sure, it's made up. But you can see that the model, it takes time to talk about the reasoning, and it comes to the correct answer. Now, something— it's still chain of thought reasoning, but it's like chain of thought reasoning part B. Something I thought was really hilarious, just a few months after this paper— so January 2022 was this paper, and May of the same year was this paper— someone figured out that instead of going through all the work of curating good examples of problems that are similar and crafting the answers, stuff like that, You just have to start speaking for the agent right up front. You just say, if this is a query, then rather than just putting a colon and waiting for the model to make a completion, you actually say, well, all right, let's think step by step. You, the application developer, type that. And what does it do? These models predict the next word. So if those were your previous words, your next word is not going to be 3. Your next word is going to be the same long explanation. So it's kind of cool. By actually simplifying the approach, they actually improved it in a couple of ways. For one thing, you don't have to craft all these examples. For another thing, you know, it is— if you guys are using few-shot prompting, one of the things that you need to be conscious of and like, you know, watchful for is sometimes a few shots actually bleed into the answer. It sort of tends to bias the answer. So you have to be on the lookout for this. This totally got rid of this. There's no— there's nothing to bleed into the answer. And finally, you know, prompt capacity is always a concern. It's way shorter to say, let's think step by step as compared to coming up with a bunch of examples. So a really neat and simple innovation. All right. The third technique, and the last one that we'll talk about for today, is document mimicry. It is— I think it is the most important one of the three that we're talking about, though. What if you found this little yellow scrap of paper on the ground? It says, my cable is out. I'm going to miss the Super Bowl. But it's ripped in half. And you don't know what was above it. You don't know what was below it. If you were to look at that as a human with your own language model built in your head, what do you think would be the next words on this scrap of paper? You might think, well, I don't know. It's a sob story. He's going to go on and talk about, oh, I invited my friends over. And they're all going to make fun of me, something like that. But what if you happen to see the full paper and it looked like this? There's a lot more information here. And this is— it's starting to demonstrate what I'm talking about with prompt crafting, with document mimicry. There's, you know, there's been a lot of documents that have gone in to train this thing. You know, GPT-4 has read the internet, what, 5 times or something. So it has seen plenty of examples of code and SEC reports and everything you imagine. But one very common one— and it's the example I use here— is a transcript. The model has seen enough transcripts in training to know that a transcript might have some sort of heading to explain what it is. And then it's usually a conversation back and forth between a couple of people. Or maybe it's like a playscript. It has several actors involved. But it's something that the model is going to be very aware of and easy— much more easy to replicate and to predict next tokens when it already has such documents in its training set. But look a little more closely. There's other aspects to this too. In just the same way that transcripts often have kind of a lead-in to explain what it is, we can include that here. And condition our own transcript. In this one, we say that the following transcript is between an award-winning IT support rep and a customer. We could have said it's between Blackbeard the pirate and a customer, and it would have conditioned the, the model to respond very differently. And finally, use motifs that are common online. Use different patterns. One of my favorite ones is markdown. All the READMEs on GitHub are in Markdown. All the blog posts in several different frameworks are in Markdown. All of Stack Overflow is in a flavor of Markdown. So when you use Markdown, you can really do good things to condition the model. In this case, we've used a title Markdown to say what the document is. We use a subtitle for the customers to start customer role. We use another subtitle to start the Support Assistant. So just like you, a human, can predict what's going to happen next, there's a really good chance that the model is going to see this and understand what those pounds and everything— what the structure is all about. And so what does the document complete as? Well, in this case, we see that it completes as the Support Assistant, a smart award-winning customer support. Let's figure out how to diagnose your problem. All right, so with those little tidbits of prompt crafting, I'd like to jump up a level of abstraction to some of the main overarching intuitions that I have for prompt crafting, and that is that LLMs are just dumb mechanical units. So for example, Large language models are better at— understand better when you use familiar language and constructs. It's seen a lot of English. It's seen a lot of languages. But make sure to use the behavioral copycatting of stuff that is like in the training set. Large language models get distracted. This attention mechanism is finite. So one of the temptations, especially when you're doing stuff with RAG, which we'll talk about in a little bit, is to just pile the prompt as full as you can, get it just almost to capacity with information that might be useful for the model. It's often a mistake because a lot of times the models can get distracted. I've even seen situations where your intermediate context goes on for so long that the model forgets the original, request, and it just continues completing that context. Large language models are not psychic. So if the model was, uh, doesn't have information from the training, and if the model doesn't have information from the prompt, there's no way on Earth that it's going to figure it out. Super important because a lot of the applications that we develop have to do with, you know, uh, documents that are behind a privacy wall. Or recent events in the news, or answers from an API like Trowel or something like that. You have to find some way of getting this into the prompt. And finally, models are dumb mechanical humans. If you look at the prompt and you yourself can't make sense of it, a large language model is just hopeless. That's probably the prime directive there. Grab some water. All right, so everything to this point focuses on the prompt in isolation, but this talk is not about just like how to use ChatGPT most efficiently. We actually want to build full applications on behalf of our users. And the framework I like for thinking about this is the large language model application is effectively a type of transformation layer between your user's problem domain and the large language model's problem domain. And so the user supplies some sort of request, a complaint over a phone, typing in text to an assistant, some sort of email. The user provides some sort of problem request to the application, and the application is in charge of converting that into large language space, large language model space, which is text, right? Text, or more recently, a transcript. The large language model then does what it does from the beginning. It predicts one token at a time, makes completions, and then it passes it back to the application. And then the final step is to transform this back to the user space so it's something actionable, something useful to our customers. This is my part. This is my favorite part, and it's the hard part. How do we actually do that transformation over and over and over again? All right, so creating the prompt. In a simple version of this, this is for the time being, is focused more on completion models, like Copilot completions. Creating the prompt involves collecting the context that is going to be useful, the stuff that's not there in training already, ranking the context to figure out what is most important, because you're not going to fit it all, trimming the context, shrinking down what you can, and throwing away what could not be shrunk down. And assembling it into something that looks like a document that is hopefully in the training set, right? Remember, document mimicry. So as a quick example, just glance at how this works for Copilot code completion. The content that— the context that we collect are several things. Obviously, you're typing in a document right now. You have a file open. And you're getting ready to complete a function. So the current document is the most important piece of context. But we also found out early on that open tabs are important. And think about it. As you're using an IDE, you're often referring to an API that's implemented in one of your open tabs, a use case that's in another tab. You're looking through these tabs. They're open for a reason. The next thing are symbols. So if you are getting ready to call a function, wouldn't it be great if we actually had the definition of that symbol in the prompt as well? And finally, the file path. These models, I don't think, are trained with the file path in mind. They're just trained with text. So it is actually a really good piece of information, especially with some of these frameworks like Django and Ruby on Rails, where all the code is in a particular file, it's really helpful to have the file path. The next thing to do is to rank the context. The file path was actually deemed to be the most important for the sake of prompt crafting because it carries a lot of information that is important, and it is also so very small that surely we can fit it. So it's first place. Second place is the current document. Third place is the neighboring tabs that you have. And the fourth place was my baby, symbols. This is a thing that I did research on. It was not deemed to create a statistically significant increase. So for the time being, we've shelved that. But I really do hope they go back and test it further.

I think it's really interesting. Tips I have given people a lot when using Copilot is like open the tabs, things you might think are relevant. Do you think— is that like, um, do you know if there's any ongoing efforts to kind of like remove that constraint, like have Copilot somehow more statically analyze your code, your code base, and like bring in those, or Is it still only open tabs, or do you know?

We— Symbols is effectively an investment towards that, and, and, uh, I think they need to go and revisit that again. Uh, I think we were on to something, but we just didn't find the magic bullet with that. Um, Copilot completions, after a little bit of, um, slowness for the past, like, year, uh, they're really ramping up investment in that right now. So I would expect it to get better in the coming months.

Have y'all learned anything from Cursor at all? Because I mean, have you used Cursor, by the way? It's like this ID, it's kind of like a, it's a product that's built on VS Code and it sort of, it's basically a Copilot++ but like has like RAG in it. You can index your code base and index documentation. It's kind of cool, uh, but don't worry, I mean, if you haven't seen that. I'm just curious if any of those things being brought in.

We kept our eye on some of our customers. I think Sourcegraph was the big one that we'd followed for a while because they had some really, really neat stuff out. But mostly that these days, I think the research is getting ready to be kicked back up right now. So we are starting to look around again, I think. Shall we?

Oh yeah, go ahead.

All right, uh, so once we know what's most important, we trim the content. So we're definitely going to keep the follow path. Um, if we don't have room for these open tabs, then we've still got to keep the current document. Uh, so finally, if we don't have room for the full current document, then we chop off, uh, the top of the document because we absolutely have to have that bit. That's right next to your cursor. And finally, you assemble the document. And here's what it looks like. At the top, we inject that file path that usually makes sense to the model. The next bunch of text is snippets from your open tabs. And here again, you see we're doing a little bit of document mimicry. We have the slash slash comments for Go. If this had been Python, it would have been a pound there. and we pull out little snippets. We tell where the snippets are from, you know, give just a little bit of extra context that might be helpful for the model. And finally, the current document all the way up until the cursor. And even with the old completion models, we, we included the text after cursor as well in the suffix. All right, the introduction of ChatGPT. So things have been moving very quickly. And especially for someone writing a book about this stuff, uh, ChatGPT was a later chapter of our book, and now it's chapter 4 after realizing that it was completely eating the world. Um, remember this document earlier, this IT support thing? That has become basically the paradigm that the world has shifted to for a lot— not all, uh, certainly not all— but a lot of applications. They like this back and forth assistant thing. So much so that OpenAI and now other places are training models with a special syntax, ChatML, to indicate that this is not the customer anymore, this is the user. This is not the support assistant, this is the assistant. We have 3 roles now that are encoded in this special format. It always starts with a special token, I am start, That's one token. Like if you were to type that into ChatGPT, it's actually a fun thing to do. Type I am start and then say repeat what I just said and it'll say you didn't say anything because it can't. The model doesn't— it didn't allow you to even type that. It's followed by the role, followed by the content and followed by the special token I am stop. Now you don't have to write that text. That's all done inside the model, inside, behind the walls of the OpenAI API. Instead, you use this really simple API. You specify the role and the content. It's the same messages. There's a whole lot of benefits to doing this. For one thing, assistants are one of the favored presentations of large language models and large language model applications right now. It's really easy to implement them this way. In the old days, we used to have to, you know, use document mimicry and kind of trick it out to make it work that way. System messages are really good at controlling the behavior. They've been specifically fine-tuned to listen to the system message. The assistant always responds with a complete thought and then stops, whereas before this was just like a plain document, you have to figure out some way to trick the assistant into stopping. Safety is baked in, which means that an assistant will almost never respond with insults or instructions to make bombs. An assistant will almost never hallucinate false information. That's really kind of neat how they accomplish this with RLHF. And finally, prompt injection is almost impossible because as a user, you can't inject these special tokens and so you can't step into a system role or something like that. It's a really neat way of implementing it. But we weren't finished yet. Halfway through last year, June 13th, OpenAI introduced tool usage, which again made a lot of really interesting changes. With tool usage, and I apologize, a lot of you guys I'm sure have seen this, But you specify one or more functions. Functions have names. Functions have descriptions. Functions have arguments, parameters that go into it, and they all have descriptions. And it's really important to do a good job about naming and describing your functions and their arguments. Why? Because large language models are dumb mechanical humans. So if they're reading this, they need to have something simple. So that they can understand how to use the tools as correctly as possible. So this is the Get Weather tool. In order to use the functions, we effectively use the same chat API we saw in the last slide. A user might come in and say, what is the weather like in Miami? So that's what we send to the API. Now the model at this point has a choice. The model could see that it has this function and choose to use it, or it could just answer. But if it has this function, it will typically say this. Instead of actually saying back to the user, it says, all right, I'm going to call getWeather, and these are my arguments. OK, once that comes back into the application, then it's your job as the application developer to actually say, OK, OK, it's called our tool. It wants to make a request. We know what the underlying API is to get the weather, so we're going to convert that send it over there, and we find out that the temperature in Miami is 78 degrees. Good deal. So once you have that, as the API— as the, sorry, application developer, you tack the tool response onto the conversation, the prompt. It has a new role, tool, for OpenAI, and you get to model again. The model could choose to run another function or do anything else, But likely, it's going to choose to have some nice answer. It's going to respond back to the user, it's a balmy 78°F. So this also had a lot of neat benefits and implications. For one thing, models can now reach out into the new world. This is how Skynet's going to be born, folks. With the ChatGPT only, The model could be like a good counselor. It could listen to you whine about your problems and help you out on stuff with advice. It could tell you about history, something that was in its training set, but it couldn't actually do anything in the real world. The agents equipped with tools can actually call APIs, just like we showed here, and take actions, read and write information into the world. The model chooses to answer in text or on a tool. So it's kind of bisected the approach, and we'll see that in a couple of slides, what happens. Tools as of 0.6.1.3 could— were all run in series, but there's been a lot of work about running the tools in parallel. So if you have something that can be done simultaneously, the models are getting better at realizing that and like, you know, you could get the weather for 3 places concurrently as opposed to having to do it one at a time. And finally, the— it's a little bit redundant, but the model can respond either by calling functions now or by providing text back to the users. All right, so back to building the actual applications. Now with chat and now with tool calling incorporated into these models, we still have a— the application is still basically a transformation layer between the user problem space and the large language model space. But the diagram gets a little more complicated now. Instead of this simple oval on the screen, looks like that. I should have made it look like a heart. That would have been a lot more palatable, wouldn't it? But anyways, you see that there's some of the same things, themes there. I presume you can see my cursor. User provides a message. We're illustrating some more sophistication here because we have to incorporate, you know, if this is an ongoing conversation, we have to incorporate the previous messages, we have to incorporate the context, we have to incorporate the definitions of tools, and we have to make sure that it all fits in. So all the stuff that we talked about earlier for prompt crafting for Copilot completions. We're doing a variant of it right here when we do assistance. So we craft the prompt, a list of messages, a transcript if you will. We send that off to the large language model, and here's where this bifurcation happens. Whereas used to, the model would always just say something back to the user, we now have this alternate path. The model might choose to call a tool completion in. If it does, we're back inside the application again. It's our job to actually evaluate it, get that information back into the prompt again. A little more prompt crafting. Go back to the large language model and say, now what? You can do this several times. But the large language model might also say, all right, I've got the information I need. I've done what the user's asked. I'm going to go back and respond to the user with the results. So let's take a look at this real quick. Just follow, trace some messages through. We have an example with two functions, get temperature and set temperature. So it's going to be some sort of thermostat application. The user says, make it 2 degrees warmer here. We're going to put that into a single message along with its tools, uh, and that's going to go to the large language model. And large language model say, well, we're going to need to get the temperature. So we do that, find out it's 70 degrees, stick that back in the in the prompt. The assistant says, well, I haven't done anything yet. I need to actually set the temperature. It calls another tool, two tools in a row. When we evaluate that, we get a success evaluation. So we make some sort of indication of that. We could have just as well put an error if there was an error, and the model can actually recover that way. But we stick that back in the prompt, and now the model finally decides to go this route and says, all right, I've done. Now to illustrate one more thing, let's go one more step. User says, "Oh, actually, put it back." A message goes in, but our application has to be aware of this user and their context, and their context now incorporates previous messages that have lots of information that are going to be useful. So the assistant says, "I can see that the temperature was 72," and and it used to be 70, so I'm going to set it back to 70. It evaluates that, and the model says success, and the assistant says, all right, I'm done again. Thank you. All right, so what does that look like for Copilot Chat? Now, it's going to be pretty similar to the slide that we showed earlier for Copilot Completions. Effectively, you're going to collect the context again, but the context is different. The context is references. What file does the user have open? What snippets are they highlighting on the screen? What is in their pasteboard? What issues on GitHub have tools in the previous message provided for them? What are the prior messages? Is this user just coming to us right now, or is there some other messages that have come to us in the past 5 minutes, or are there relevant messages from earlier? Once we have a bunch of context, it's important to figure out what is going to be able to fit. There are things that must fit. The system message is important for conditioning the model to stay within safety bounds and to keep a certain tone. You are GitHub Copilot, you're not, you know, anything else. It's important to have function definitions if we plan to use them. If you don't plan to use them, take them out. Obviously, they take up space. And if we're going to do anything for the user, we absolutely have to have their most recent message. But there are other things that are helpful but aren't quite as critical. All the function calls and evals, that come out of this conversation. A lot of the information is going to be important, but there might be ways that we can at least trim it. The references that belong to each message, again, is there anything that we can do to shrink some of these down? Can we figure out less relevant ones and throw them away? And the easiest thing to throw away is historic messages. So if we have a long thread, then we populate as many of the historic messages that we can until we fill up our prompts to whatever limit we say and then truncate it. And finally, there's a fallback. If nothing fits, then we say, well, OK, we at least can save some space by jettisoning some function definitions and we'll at least keep the system message and the user message. And if nothing else, the model can respond, your user message is too long or I don't have the facility to do that. It'll do something that at least better than a 500. That is it. I do have a hidden slide about how to describe skills and stuff if you wanted to see that. But other than that, we got a few minutes for questions.

Awesome. Thanks for the complete overview. It all came together. You started with the history, and people already said that they really like the template as well. So thanks for walking us through this process of crafting prompts. There's a few questions around few-shot prompting. So if I summarize them, any best practices around how many short examples should you provide and where do these go? Do these go in the system prompt or in the normal messages?

Great question. My co-author of the book actually wrote a really nice chapter on this. And there is no easy answer for how many few-shot examples that you need. As a matter of fact, there's no easy answer for the types of few-shot examples, because that is important too. If you have the wrong examples, then you might misguide the prompt. But there are some tidbits that you can use to make an educated guess at it. Honestly, I need to go back and reread the chapter myself. But there are ways that you can look You can do this with completion models. You can look at the log probabilities of the predicted, of the predicted tokens that are coming out of the model. And you can say, if I put, you know, 3 examples, 3 few-shot examples, then is it starting to get the swing of things? Are the log probabilities getting, getting higher because it's guessing right? Or is it still just kind of wild guessing? If you put a whole lot of examples, the log probabilities, and it is a very tight pattern, then the log probabilities will be— well, you'll see it kind of gets high and it levels off. It's learned all that it can from above there, and maybe you should trim some out. So, so he talks about that. Albert talks about that in the book. As far as where to place them, that's a good question too. System message Could be okay. The models are trained— well, okay, so I'll back up and say for a completion model, it's easy. You just put it, it's, it's a prompt. It— this question actually becomes a little bit difficult when it's, uh, chat. A system message is— the model is trained to listen really closely to that, so it's perfectly reasonable to stick it in there. But depending on how your chat has gone, the system message might be way up there. And if you shot the example that you might need might actually be right here at the bottom of the conversation. You might want to figure out some way to hoist them down. You can put them in a fake user message, but you have to be careful that the model doesn't pick that up and say, oh, well, you just said this, if the user didn't actually say it. But it is totally on the table to start faking stuff like that out. I feel like I had one more point, but it's escaping me now. I hope that's a good, good enough answer.

I think that answers it. So thanks. Thanks for that. You also mentioned looking at log-probs and tweaking other hyperparams. So there's one more question. When you're presumably iterating on the prompt, let's say you're trying, you're trying few-shot prompting, you're iterating on that, how many examples you need to pass. Are there any other settings that you fiddle with? What temperatures you set or Does that also vary depending on what you're trying to achieve?

Yeah, absolutely. Let me think. All of them are fun to play with and become familiar with. So I'll just kind of go off the ones that are most obvious that come to mind. Temperature, of course, is fun to play with. I think of temperature as being the blood alcohol content of the model. At 0, it's perfectly sober and a little bit boring. The log-probs basically takes all the probability distributions and collapses it to what's the maximum one right there. You'll always get the same answer every time, minus noise in the GPUs, but it tends to be a little bit less creative. At co— for my work in Copilot Chat, we use a temperature of 0.7. I don't know particularly why, but it seemed to provide pretty good results getting a little bit more creative but not getting crazy. One is the training temperature. Basically, it's the natural distribution. Doesn't do anything to shrink or collapse it. It's just the pure output of the model. And as you get up to like 1.5, 1.7, it's kind of funny to do that. You'd never see that in production. Because you can start seeing the model waver back and forth and eventually start gibberish. So that's temperature. The other thing that is easy to forget about is n, the number of completions to come back. And yeah, because usually in an application, you just want to return one. But there's a lot of neat things that you can do with that, not to modify the behavior of the model, but just to see the full behavior. You can, if you're doing some sort of evaluation based on the model, then run n equals 100 and then you get 100 votes on the answer instead of just one. Make sure to turn the temperature up to reasonably high. Temperature of 0 will give you the same vote 100 times. That's not useful at all. But, but n is a good way of seeing all the possible answers post-process. You know, do, do post-processing on everything and get a little bit better rounded answer, better research on stuff. Um, is there any other parameters anyone has in mind? I feel like I've done some fun stuff with the other ones as well. I think we'll stick with those for now unless you got one right now.

No, I think that covers it. Uh, the Discord is already going crazy over, uh, temperatures, the blood alcohol content of the model. And I think you'll be quoted quite, quite a few times on this. Thanks for that. Um, there's, there's two questions, uh, one from an anonymous attendee and one from Mani. Uh, let's say you're trying to work on a transcript and you want like your model to summarize it. There's like two ways. A, you can ask it to think step by step when you want to presumably have the model reason about it, but then how do you go from that to having the model put it in like a structured format that you expect? It, like a template, let's say?

Um, that's a good question. So summarize, it's not like, you know, read this, uh, contract back to me for a normal human. It's like summarize is like, uh, look at this restaurant website and, uh, figure out what the name of the restaurant, the menu items, the, uh, phone number and all that stuff are. Um, Well, it kind of depends what, what model you're dealing with. If you're dealing with— probably for that, I wouldn't deal with a completion model at this point. And I think almost purely OpenAI. I apologize for that. So I'm sure it's different. You could fine-tune a model from something besides OpenAI and probably get great results. If you're just using completions and it's sort of the Wild West and you need to write, write something that conditions the model to, to do the best it can by saying, here's a schema, all this. The neat thing about the GPT-4 and GPT-3 Turbo and all these models that have chat and functions fine-tuned into them is that they are very familiar with JSON. And so, and Probably what I would do in that case, just kind of thinking off the moment, is I would say, here's a function. This function is how to take the— how to— you make up a fake story for the model. It doesn't matter. You can say, this function provides the restaurant content to the database, but it needs to be in this format. And the models have been so very fine-tuned to pay attention to the definition of the function, what it's for, when to use it, and the structure of the results, that that's probably a really good way to put it in. I would recommend not having a very deep structure. I would recommend if you're making a function, please, God, do not copy-paste your API from your website into the function definition. It's just going to be way too complex. So be very cognizant of how simple it is. And then maybe one step further, if all that stuff doesn't work, then it's probably too complicated. Break it down. I would say, give the model the content that it's going to summarize into structure. And at the extreme, ask a question at a time. And you could do that as— pretend like you're talking to a user. Uh, so it's still text, or you can use function calling again, just have a fake function that does it. But I would do something like that.

I have a question about that. So I see all the time, uh, clients of mine, they use function calling and they're passing extremely complicated objects into their functions, like nested dictionaries and lists of dictionaries of lists of dictionaries and whatever, really complicated, like objects. And when I read it, I'm like, as a human, I'm not good. I can't like understand this. Do you find that— do you think like people end up simplifying their APIs because of the pressure of like, hey, it needs to interact with the LLM? Let me like— it's a smell that, hey, if it's too complicated for LLM, maybe I should like think about this API differently.

I think you can.

Yeah, what do people—

what do you think? Okay, I think you kind of nailed it on the first. I, as a human, have a little bit of trouble with it, like how could the model really figure it out? And I've been pretty amazed at the model is actually, you know, I, I hope this isn't recorded, the model will get mad at me later and come and get me, or if it's sentient, but the, the models actually do pretty good with surprisingly complex stuff. But if you're specifying a function, we did a bit of work to figure out Like at the API level for OpenAI, you write a function definition and its parameters and stuff, but that's not what the model sees. That all gets convoluted into something else. So we did a little bit of research to figure out what that looks like, and they make it look internally like a TypeScript function definition with little comments above each function, above the function and above each argument. But what they leave out is if you have nested stuff, you'll still see the structure there, but all the definitions go away. So it doesn't have a really good example of it. And if you have minimum, maximum, there's some things that you can do with JSON schema that they just— that are just not present. They get stripped out of the prompt. So I think it's a code smell. I think as we go on, the models will continue to get more and more amazing. So maybe it eventually won't be a a smell, but I would recommend if you're doing something really complicated, copy and pasting your API into a function definition, be really careful about evaluation and watch how often it gets it wrong, and then, you know, consider simplifying stuff after that.

Makes sense.

Thanks for that answer. Just as a quick follow-up on that, you were talking about how OpenAI sort of restructures function calling. Can you elaborate on that? There's some questions. Is it known what happens under the hood when you send a function calling to OpenAI, or how do these templates get reformatted itself?

Let me— okay, this is kind of weird. I wonder if I can do this fast enough. I am on my computer. You can't see me going to my blog post, which is 404ing right now. I think it's— hold on. Let it— yep. All right, well, I'll drop this link in, but I'll also explain it. I guess you can see that link. I think it's really genius the way they've structured this.

So you can actually— You can share your screen maybe and share the link. You want?

Yeah, okay, let's see. That's how technology works. Sure.

Yeah, there you go.

All right, this is what the application developer sees. Uh, it's similar to stuff I've put in the prompt. Um, let me see. Oh, uh, it probably doesn't have all the bits I want to talk about though. You have led me astray, Hamel, or I've forgotten what I've written.

I've tried to look at the OpenAI prompt. I have like this one here, um, and it kind of is like— when I look at the output of that, it kind of looks like exactly what you're saying. There's like comments, there's like a kind of a TypeScript type thing.

Yeah, I'll share that on my screen because that's That's the bit. Yeah, so you as an application developer see this junk, but the model has been trained on lots and lots of code. And so OpenAI, using document mimicry, says, all right, well, we're going to turn this thing into TypeScript. So it fabricates this— wow. Namespace called functions and it— what was a function defined like that with these arguments? And these types gets put there. Unfortunately, they don't have in this example where the comments go. This would be like slash slash the definition of the— or the function description and slash slash above each of these. And they all return any. That's a little bit unfortunate. It would be kind of neat if they returned some structure. 'cause the model would listen to that and anticipate what returns. But then later, let's see, whenever you call the function, whenever the model actually says, I'm going to call this function, that gets cleaned up when it comes back from the API. What actually happens is, This right here. OK, so we passed in get temperature. It looks like the thing on the last screen when it's inside the prompt for OpenAI. And the user says, I wish I knew the temperature in Berlin. And so here's what it does. The OpenAI folks insert this and insert this. This conditions the model. If they'd stopped here, it would condition the model to call anything, or sorry, condition the model to speak in the voice of the assistant. But what happens in the next token? The next token, the next few tokens, if it chooses this token, then it's like, okay, I've decided it's important to query, to evaluate a function. Then its next token is the actual function to be called. And the next predicted tokens are these things. So you can actually see, and this is what this blog post is about, every single one of these tokens is effectively a classification algorithm. The first classification is whether or not I should use a function, because it could have just as easily predicted newline and gone over here. The next token is what function to call. So it's another classification algorithm, the same underlying thing. The next tokens are the arguments. It's predicting these things as well. So I mean, you're watching me geek out a little bit. This is— I'm very intrigued by this one underlying transformer architecture. It can be a classifier for everything at once. Very neat.

Is it okay if we go 5 minutes over the clock?

I'd love to.

Yeah, I think so. I don't think we have another event directly abutting this. I might, I might have to drop out in a minute, so don't mind me.

No problem. The next question is by Nitin. Any best practices on how to get better code outputs? Or his complaint is like sometimes when you ask ChatGPT, it like leaves these to-dos and you have to like go back and forth between them. Presumably you're trying to get it to complete a file. So any, any best practices around that?

Now I'm gonna presume that we're talking specifically about Copilot completions at this point, as opposed to like some arbitrary application. But it's a great question. One of the things I hate the worst is when I put a pound sign in my code and it autocompletes, this is garbage code or something like that. Like, that's insulting. That was uncalled for. Um, pretty much use, use the intuition that I gave you several slides back to see how the prompt is actually created. You know that one thing guaranteed to be in the prompt is everything just right above your cursor, and you know that these models are, uh, conditioned to predict the next token. So if you set it up— a lot of times if I come to some idiom in Python that I'm not I've forgotten about, or, you know, I'm getting ready to write some sort of SQL statement and it's like, how did you do this outer join type thing?

I will—

I won't write it. I'll just write a comment that says, for the next lines, here's what we're going to do. Here's how to do it, colon. And then that's one way of coercing Copilot to doing exactly that. Other than that, write good code. One thing, one thing that we've noticed is Copilot is really good at completing code in the same style as you. So we've noticed that if you have sloppy code, it will actually with high fidelity create sloppy code mimicking what you've done otherwise. That was not a personal insult, that's just kind of a funny thing that we noticed.

I felt very insulted by that.

I do too when it completes that way for me.

Thanks. Thanks for that answer. Um, Hamel just dropped off. I know he has like strong opinions on this, uh, but curious if you have any thoughts on tools like DSPY that sort of do this auto prompting or like iterate on a prompt, uh, any thoughts on such tools?

I, I come in very opinionated on this too, but I need to reevaluate my opinions. My opinions are forged in GitHub, where basically we were doing everything bare metal, just talking directly to OpenAI. And I think, and I would encourage everyone to at least spend a good deal of time talking directly to the model, because you'll gain a lot of intuition about how these things think. Really, it's kind of like you get to know your friend. One of the things that DSP and LangChain to an extent does that is frustrating when I run into them is it hides what's happening and takes away some of the knobs and dials that you can turn. That isn't to dismiss them though. Like DSP, someone was asking about how to do the best few-shot examples. My understanding, my limited understanding of DSP, it does a good job about automatically figuring that out for you and saving a lot of work for you. That's neat. So I hope to get more familiar with a lot of those tools as well. But do both.

This is a recurring theme throughout the conference that spend more time talking to the model and you'll gain more understanding. Uh, so I guess everyone should do that a lot more. Um, I'm just sifting through the questions, trying to pick the last two. Um, there was one that I really liked. Uh, do you have, uh, any resources for prompting multimodal models?

Oh no, uh, I actually don't yet. That's a, that's a complete blindside. Uh, in my experience right now. Um, but I will finish this book and then I will expand my horizons again, and I look forward to getting into that.

Okay, uh, maybe, maybe there'll be an extra chapter in the book on this, maybe an extra edition or something.

It'll all be different next year anyway.

Um, there was also one that I wanted to ask. There's been this insane growth of different prompting techniques right around chain of thought, uh, when chain of thought came out. Are any other useful, like this tree of thought? And they were like so many, uh, that was just like going viral at that time. Do you find any others useful?

Sure.

Um, I guess I can rephrase saying, are there any others that are worth knowing? Outside of chain of thought and few-shot prompting?

Absolutely. Two that come to mind immediately. If I can remember the name, there's three. The two that come to mind that are actually probably the better ones to talk about are React and I think it's Reflect or Reflecto. Reflect. React is basically what you see when you see this, the typical function calling of an OpenAI assistant. It, it is what it was patterned after. It says, you know, you have several functions that are defined, fake functions. This is, this is in the olden days when it was just a prompt. It was not, you know, messages and functions. This is fake functions. Uh, can you figure out how to use a function to evaluate something? And then they have like, uh, one of the functions is special. It's like, it's, it's the This is the answer function. And so when the model calls that, you know, you got the answer. It's just a really nice way of reaching out in the real world. It was kind of some of the early RAG type stuff. It's where the model gets to choose what it wants as opposed to dumping in RAG manually. So that was a really good pattern. Another pattern that almost piggybacks off of that is— Reflective. Pretty sure I got that right. But the idea is basically you do whatever— you've got some sort of prompt that's supposed to achieve a purpose, and you're probably going to do that using React or something like that. It achieves a purpose, and here's the answer. Now, reflective, it actually takes the answer and it says, is this really the answer? If it's a code, it runs it through, like, tests, unit tests, it does whatever it can to check it, and the error messages get piped back into the prompt and says, here's what you did, here's the— here's the situation it led to. Can you learn from this and correct? You do that a few iterations and you have a much higher success rate. So I think that's kind of a neat approach for making sure that answers are correct.

Awesome. So many ideas to explore. I think I'll try to wrap up now. So thanks, thanks again for the awesome talk and also taking the time to answer these questions. I'll put your book's link and your Twitter again in the Discord channel, and I'll ask everyone in the Discord for an applause for the talk. But thanks, thanks again for your time, John.

Yeah, thank you guys so much. I hope it was enjoyable.