Delving into The Prompt Report, with Sander Schulhoff of LearnPrompting.org

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thorenberg. Hello and welcome back to The Cognitive Revolution. Today, I'm excited to welcome back Sander Schulhof, creator of learnprompting.org and organizer of the Hack-a-Prompt contest that we discussed when Sander was last on the show back in January. This time, Sander's here to talk about the recently released Prompt Report, a mammoth 78-page survey paper covering the current state of prompting techniques for large language models. As AI capabilities have exploded over the last couple of years, so too have the number of papers exploring how to get the most out of these models through clever prompting techniques. Sander and his team took on the Herculean challenge of reviewing and categorizing this vast literature to create a comprehensive taxonomy and guide to the field. In today's conversation, we delve deep into some of the key findings and insights from this work. We discuss best practices for few-shot prompting, the challenges of ensembling and evaluation for open-ended tasks, multilingual and multimodal techniques, the current state of prompting for AI agents, and even how automated prompt optimization systems like DSPy can outperform human prompting experts like Sander, a result that I personally can't help but see as a sign of things to come. Beyond the technical details, Sander also shares his experience leading a large research team and offers reflections on trust, testing, and project management that I think will be valuable for anyone embarking on a similarly complex technical research project. As always, if you're finding value in the show, we'd appreciate a review on Apple Podcasts or Spotify, or you can just share it online with your friends. If you have any feedback or questions, you can always reach us via our website, cognitiverevolution.ai, or you can DM me on your favorite social network. Finally, for now, a quick update on our jobs board. I'm glad to say that I have made the first of what I hope will be many very high-quality connections between someone looking for a new role and a very interesting startup. In this case, the startup happens to be in stealth mode for a little longer yet, but I hope to tell you more about that soon. And in the meantime, I hope that encourages you to submit your resume if you're interested in a new opportunity and haven't already done so. Now, I hope you enjoy this wide-ranging discussion on all things prompt engineering with Sander Schulhof of learnprompting.org. Sander Schulhof, creator of learnprompting.org, previously here to talk about the Hack-a-Prompt contest and paper. And now returning to talk about the Prompt Report. Welcome back to The Cognitive Revolution.

Thank you very much.

So you are prolific. I wanted to start off by just talking a little bit about what you do and how you're doing it all. You've got the Learn Prompting, which was originally a project. Last we talked, had kind of evolved into a startup, and then you've put together a couple mega papers along the way as well. So What does the Sander Schulhof portfolio look like these days? What's the status of learnprompting.org as a business? And how are you finding time to bring these teams together and do this deep analytical work at the same time?

Yeah, so starting with Learn Prompting as a business, we are currently building out our generative AI course offerings and building up the team, actually looking for a first full-time first hire. At the moment, something of generalist to help with a bit of video creation, a bit on the business side. And recently I've been experimenting a lot with automating content creation for like information teaching. So there's a lot of YouTubers out there. There's a lot of content on Coursera, all these other platforms, talking to some C-suite execs at like consulting firms. They often approach for enterprise deals and see content a couple years out of date, and it's really painful to fix that content, and companies don't want to buy that out-of-date content. So we currently have a way of keeping our content up to date always by building these sort of videos that we can compile in a way by, you know, say we have a script we use. like 11Labs to read over the script. And a couple of months later, we get feedback from our customers, change the script, one Python script, and the whole video is updated. So looking at a lot more of a reproducible way of doing content creation there.

Interesting. And that's a technique you've developed as part of Learn Prompting's development?

Yeah, that's correct. So there's a lot of complicated tool pipelining you need to put together. Unfortunately, there's no way to just say, to an agent, hey, make me a video about prompting, cover these topics, go to these websites and do tutorials on them. That's all quite hard to set up at the moment, and we're building the infrastructure to do that.

Yeah, interesting. I mean, that's definitely a startup in its own right if you wanted to turn it into it. Cool. Interesting. So heavy use of LLMs. That's something I wanted to get into a little bit in the context of this prompt report on a meta level too. Before we get to the current work. Um, are there any updates in the Hackaprompt world or any other new techniques or vulnerabilities that you would update us on since last time?

Not really. There are a couple niche things coming out about, uh, attacking different systems and companies, but I haven't seen any like huge new techniques come out. I think the Azure CTO put out Skeleton Key. But it's relatively similar to a lot of other attacks that we've covered. And actually with that, there was a bit of an uptick in the debate over what constitutes prompt injection versus jailbreaking. I can send you like a tweet interaction there after, which is really interesting to read. And actually I'm in this like security Discord where a lot of those prompt security people are and It has become something of a hobby to argue about or discuss the definitions of these terms because I'm pretty like set on prompt injection involves like developer instructions in the prompt as well as user instructions, where jailbreaking, it's just the user and the model. But there's, it's like a surprising amount of discourse behind the scenes on that.

Interesting. What's at stake in that? debate? Like, why does that distinction matter or where does it become operative?

Good question. I mean, honestly, it's not like an incredibly important distinction for me. It's just a matter of, I guess, one, clarity, and two, following the sort of definition of the person who originally proposed it, which was Simon. And I actually have a blog post on Learn Prompting about how My understanding of the definition changed over time. I like used to believe one thing, and then actually Hackaprompt came out and Simon put up a tweet that was criticizing the definition we used. And I went back, read what he was saying, and I was like, yes, we were wrong. So I wrote a little reflection on that, updated the paper, and it's pretty interesting, I think, seeing the, my thought process change over time, which you can read in the blog post.

Okay, cool. Things that have caught my attention, I wonder if you have any thoughts on them. One is the Anthropic mini shot. I don't know if that's, I guess, I think more of a jailbreak is how they framed it as opposed to a hack prompt, but you could maybe correct me if I'm wrong there. Also, I follow Pliny, as I'm sure you do, who is doing all kinds of crazy stuff, which is, that would be Prompt hacking, right? Because he is working through like consumer interfaces, which means there's presumably a system prompt. He's exposing the system prompt as one of his main exploits, right? So.

Right. I would, from a technical perspective, with like Simon's definition, I would generally consider what he does to be prompt injection, as there's often a system prompt involved. He and a lot of other people call it jailbreaking generally. He does a lot of really impressive stuff, or I guess I should say they do a lot of really impressive stuff. And as far as the many-shot prompting goes, I guess I'm not as impressed by that because we saw a lot of techniques like that during Hack-a-Prompt where people would put in a bunch of few-shot exemplars showing what they wanted. And many-shot is just many few-shot exemplars, or many exemplars rather.

How about on the mitigation side? It seems like the OpenAI paper on the hierarchical instruction following is sort of really meant to try to address this, right? Where they sort of define, like, the system prompt is the first thing, and then you can follow the user prompt, and then there's sort of this cascade of, like, what's most important, which is pretty intuitive. But actually training for that and trying to get the system to respect that consistently, obviously, is where the rubber hits the road. When I saw that, I was like, oh, this seems like they've made very substantial progress. And then when I watched Pliny's work and have read some of his recent analysis, he has said that he feels that it's getting easier to prompt inject/jailbreak the systems as the models get more powerful, maybe because there's just more surface area to attack. Like, he's doing these weird things now where he's embedding prompts into images in some cases, and He just seems to be finding soft spots in the defence that people haven't anticipated. What are your thoughts, if any, on the hierarchical instruction following? And how would you characterise the sort of offence versus defence dynamic in control of these systems over the last 6 months, let's say?

Not really any particular thoughts on specific defences. Actually, one thing I was interested in recently was Lacra. I think they ran a competition recently and had like an adaptive system prompt where whenever someone got by it, they would use that like prompt injection prompt to modify their system prompt and thus improve it. So kind of an adaptive defense over time. And I think that's a super interesting approach, but I think defense here is always a losing game, unfortunately, because just for whatever reason, we can't. understand the models enough or can't control them enough to ensure that they won't have this adverse behavior. So when you look at someone like Pliny or, shoot, this other company—

Haze Labs.

There we go. Yeah, so I did a bit of beta testing with them and I talked to them like a year ago or something. Didn't really believe that they had automated jailbreaking Actually saw their product in use, believed it a bit more, actually used their product, was extremely impressed because it was able to perform jailbreaking for an arbitrary intent, like give me some bomb-building instructions or racist output within seconds. And I really did find that to be incredible. And they were the first people I ever saw doing that. So really quite impressed with that. company. But overall, defending here seems to be a losing game. I really don't see a lot of promise in most of the techniques coming out, like training another model to detect it. I really think this is a problem that's going to be solved by the model creators themselves, some part of training, some architecture switch. I think it's going to be solved with the model developers at that level.

Does that suggest that things like the Anthropic Golden Gate Claude runtime feature detection are where you're thinking? Another interesting one I'm sure you've seen is from Dan Hendricks and a bunch of co-authors on short circuiting, where they basically, again, look at the representations through the layers and try to identify when something harmful is emerging and to kind of reroute that to a refusal. Is that sort of the track that you're expecting to work?

I have no particular opinion or guess on the track that it will go down.

Okay, fair enough. Well, that was all catching up on previous topics. Let's get to the prompt report. So this is a 78-page monster of a paper that's really a review of probably hundreds, if not maybe even into the thousands of papers that have been published over the last 18 months or so as prompt techniques broadly have proliferated. Interested in how you brought together the team to do this, because you brought together a pretty substantial team, and also any techniques that you used in terms of pipeline language model assistance to kind of corral this massive amount of information. And then from there, we'll dig into the actual taxonomies you've created and all the finding.

Yeah. All right. Start with the team. Actually, I'll start a bit before that, like kind of why this happened. So I was doing research under Professor Philip Resnick and I wanted to run a, like, AI-human, like, interaction study where basically I would try to measure how much more efficient AI would make humans at a given task, generative AI in particular. LLM, say ChatGPT. And I spent maybe a couple months learning how people do this and writing up reports. And at this time there were, I think, about 13 different studies analyzing how LLMs help people be more productive and giving hard numbers. Some of the n's were quite small, like 11, but you also had a couple thousand people studies. And so I thought— Why not do something like this? But after I got my sort of research done there and put a little report together, I realized that I really didn't have time to do this. It would end up being like a 6 to 9 month project, and I needed a pivot because I needed to basically deliver something for the semester. And I thought, well, you know, I've always sort of thought about and talked about doing a survey paper for prompting. How hard could that be? And so I gave myself like a 3 or 4 month timeline. I was like, okay, I'll go to my NLP lab clip here at UMD. And also I put like a little ad in the Startup Shell Discord. Startup Shell is a student-run startup incubator at UMD, a lot of very talented people there. So got a few people there and then we started doing like weekly meetings. Talking about what kinds of things we want to analyze, how we could use AI to do it. A couple more people started to get involved. Shyamal from OpenAI got involved, reached out to him, and then talked to some old advisors of mine. They got involved, and I was reaching out to, you know, as I was reading through the literature, I was talking, I was emailing authors of papers and saying, hey, you know, I really like the work you did on this paper, love to have you Come join us on this very massive project, and got a couple people coming through there. And then closer to the end of the project, we had 4 suicide experts join as authors. And this was related to the case study at the end of the paper, which is focused on detecting suicidal intent in social media posts. And I think that is pretty much how everyone came together. And then I think your next question was about the tech side of things. So when the project started, a lot of it was, I personally think I know pretty much every prompting technique that exists. So I was going through the paper and like, you know, making a bunch of sections for every technique I wanted in there. And I was just reading through every prompting paper basically I could find. There were some kind of smaller survey papers or specific to different domains. And at this time I was trying to figure out what sections we wanted in our paper. And we ended up with like a main text prompting and then multimodal, multilingual agents evaluation. Those ended up all being quite core. And so at the beginning, I kind of developed this list basically off of how I felt. So somewhat arbitrary. And I got the recommendation from some of the project advisors to do topic modeling to figure out what topics in the space were actually important. We did that and funnily enough, they actually aligned with what I had already selected as the topics for the different sections. So that was good. And then looking at pipeline development, we explored like, oh, can we get GPT-4 to go and read like every paper on arXiv and see if it's a prompting technique or not? And that ended up being kind of complicated and very costly. So we did a keyword search first with like 44 different keywords. Across arXiv, Semantic Scholar, and ACL, pulled all those papers, deduped them. We did a little bit of human review, so reviewed about 1,000 papers and said, okay, they're in or they're out according to some survey questions, which were like, oh, does this method require fine-tuning? If so, it's out. And so we had humans review a bunch of papers, and then we used that dataset to prompt a model to review the rest of the papers. And what that allowed us to do was throughout the paper writing process, we could keep updating our dataset by just pulling the most recent papers and having the AI read through the ones we hadn't reviewed and decide if they're in or out. And we had decently high accuracy there where it was a worthwhile thing to do. That was pretty much the data pulling pipeline. It was quite a pain to set up and like keep clean, and the pipeline itself takes like a couple hours to run. So you have to be kind of careful while you're testing it not to run it and waste a bunch of time and money, and then it has an error or something. So put a considerable amount of time into that. Actually, my co-author Michael Illey really led the development of that quite strongly.

Cool. That's really interesting. I'm glad I asked you about the techniques. Hey, we'll continue our interview in a moment after a word from our sponsors. I just did a Ctrl+F through the paper and confirmed that the word delve does not appear even once. Did you remove all the delves from the ChatGPT output or what? To be honest with me.

That is a really good question. I did not explicitly do so. So I should say that all of the paper writing is human done. There, people likely used ChatGPT or whatever to some small extent while writing. And I will say, you know, this is something you really need to be careful with when you're running a large research team. There were times when I was reading through and I was like, ah, someone used AI to generate text. And, you know, I don't really care about that. It's not a problem. The problem was that they did it badly. And so the wording didn't really make sense. And it was kind of obvious, you know, you could tell. So I performed at least 5 passes of the 70+ pages to remove all of this stuff. Just sort of over time, I would do pass and pass and pass updating stuff. But my final pass, I was like, okay, I'm going to read every single word and make sure there's nothing weird here. Someone probably used the word delve and I removed it, but I never did a Ctrl+F to find all the delves. I'm actually Very surprised that there are no Dells. I would expect there to be at least a couple.

That's very funny.

Okay, cool.

Really good insights there. So coming to the results then, I kind of want to just start at a high level and then go down some of these particular kind of use case or smaller scale areas. Let's start off with just the overall taxonomy. I've got the graphic in front of me. People can pull up the graphic and look at the hierarchy there, but How do you sort of describe this using your own heuristics, your own mental models?

Sure. Good question. So whenever I think of prompting, I think of, I guess this is more in the sort of advanced category of prompting because you often need code or some kind of pipelining to build a lot of these techniques. I just kind of naturally classified things into different domains. You have your few-shot prompting. Your chain-of-thought prompting. Actually, a lot of people think chain-of-thought is, oh, it's just like one thing, chain-of-thought. But no, there's actually a ton of really super specific chain-of-thought generation techniques. And then you have stuff like decomposition, where you're breaking the problem down into usually multiple subproblems. There was actually a lot of debate about whether to combine the decomposition and chain-of-thought thought generation sections into one. And the argument for that is kind of like, okay, chain of thought, you're prompting it to generate its thought and naturally it like breaks the problem down in doing so. And so it's kind of decomposition. And then there's also a lot of decomposition and chain of thought techniques, which are kind of similar and some bridge the gap where it's like, okay, write out all your thoughts and break the problem down into subproblems. And, you know, that's your thought generation prompts. And how do you classify that? Sometimes we had to make these slightly arbitrary decisions where it could have gone either way. And this is really just one way of organizing these techniques. I could imagine another where you have like a very different type of graph where each technique is assigned to multiple possible categories. And another example of this is few-shot combined with Chain of thought. Lots of techniques doing that. Really a natural extension for many techniques, even if not done explicitly in the paper. So there's a lot of kind of confusing stuff, how to categorize stuff. Then you have a lot of zero-shot things like role prompting, style prompting. And then let's see, Ensembling, when you have kind of multiple prompts trying to answer the same question and then you usually use the majority result. And then self-criticism, when you get the model to criticize its own output and then usually improve it. Those are kind of all of the really fundamental classes of advanced techniques that I found. And then there were even more advanced techniques that we left for later sections, like the agents and eval section.

Okay, cool. Yeah, it's almost like you need a— this is probably just because I have sparse autoencoders on the brain, but it's almost like we need a Sparse autoencoder for the different prompt techniques to then say which of the techniques is active in any given prompt, because certainly they do recombine in all sorts of different ways. I guess high-level highlights, anything that you've found to be like diamond in the rough kind of discovery or anything you think is underrated from a prompt standpoint that people You know, and you can assume with our audience, like they're going to know few-shot techniques. They're going to know like chain of thought, but kind of going into that second or deeper levels of your analysis, was there anything that jumped out at you as being like, more people should know about this?

Yeah. First thing, we went super in-depth on few-shot. There ended up being a lot of research there and also a lot of stuff that we could clarify there. So one thing is like few-shot and in-context learning is not at all the same thing. When you prompt ChatGPT and say, tell me a story about a dinosaur, that is in-context learning technically according to Brown 2020, because, uh, in-context learning is a form of task specification. And the idea here is like in the past we'd train models to do a very specific task, like classify Reddit posts. As positive or negative sentiment. Now we have these LLMs that do any task. All you have to do is give them the task and they do it. And the process of you giving them that task to do and them doing it is novel. And so in Brown, they call it in-context learning, but they also consider few-shot prompting to be in-context learning. So there's a lot of discourse in the community that conflates the two. We even found other research papers who had read through Brown et al. and still conflated them, which was—

And that paper is the Language Models Are Few-Shot Learners. That's the original GPT-3 paper.

Yeah. So reading through that paper, I probably read through the paper actually 20 times or more. And talking to Shyamal from OpenAI about it, and, you know, he was able to ask internal questions to the team, was super helpful as well. But besides the definitions, we were able to pick out 6 key pieces of advice when designing a few-shot prompt. And this is a diamond that I wish I had like a year ago because we really go through kind of all the things you need to consider. How many exemplars do you want? Does the order of the exemplars matter? I think that's super surprising to people that, like, depending on how you order the exemplars in the prompt, you can be down to 0% accuracy, up to 90% accuracy, really all over the place. So that is like a super surprising and honestly quite frustrating thing to realize because it feels so arbitrary and there's no way to figure out what's optimal.

So can we be a little more explicit about this 6 point thing. Is this section 2211, few-shot prompting design decisions?

This is, yeah, effectively. I'm referring to figure 2.3 in particular.

Okay. Yeah, exactly. Perfect. So you want to run through that in just a little bit more detail? Let's take our time on the diamonds. All right.

So we have 6 pieces of advice, basically. Each piece of advice could hurt accuracy, but generally improves it. And I just sort of say that as a warning where these aren't guaranteed to give improved results, but many times will. All right, so first one is the quantity of exemplars. It generally helps to have more exemplars. I try to put as many as is reasonable in my prompt, kind of weighing like cost with accuracy. And you'll also speed. And then exemplar ordering, we recommend that you randomly order the exemplars because if you put like all of the positive exemplars first and then all of the negative ones, the model might be biased to choose a negative one next because it's just seen so many negatives in a row. And then you have label distribution. You want a balanced label distribution, which basically means if you're doing binary classification, you want an equal number of exemplars from That being said, I put an additional note here, which we don't have in the figure, but do have in the paper. If you have already imbalanced data distribution that you're trying to predict, say, you know, like 90% of classifications are going to be positive, that's just the ground truth. If you include a sort of a 9 to 1 exemplar ratio, that can boost your accuracy overall. So generally shoot for completely balanced, but depending on your data distribution, you can change that. Next thing is exemplar label quality. Ensure exemplars are labeled correctly. The reason why we mention this is because, one, there's a lot of research that suggests that even if you input incorrect labels with your exemplars, it doesn't hurt accuracy all that much. And then the second thing is people often sort of auto-generate prompts from datasets where they have exemplars in the datasets that might be mislabeled. And so if you really want to make sure you have the, uh, the best chance of getting good accuracy, you want to make sure those exemplars are properly labeled. And then the next thing is exemplar format. So a lot of the time when I put exemplars in my prompt, I do like a Q: input, enter, A: output. So that could be like question Q, I'm so mad, answer A, angry, if I'm doing binary classification of sentiment, something like that. And so we say choose a common format because reading this paper that talked about prompt mining, I don't remember the original paper, but we have it in our paper. They found that by reading through a corpus that the LLM was trained on and finding common formats of sort of questions being asked and answered, If they use that in their prompt, they were able to improve their accuracy. And I think that this has something to do with sort of— let's see, how do I explain it? Do you know the idea of if you have a smooth loss function, generally you can get better results when minimizing it? Yes.

Okay.

So I think that there's a similar thing that happens where the space in which you ask questions is smoother if the LLM has seen that format in the past than it would be otherwise. So if I say like, question, 10 equal signs, I feel sad, answer, 10 equal signs, angry or sad or negative, the LLM probably hasn't seen that format before. So it's probably going to be a bit confused or a bit more confused as to what to do there. But anyways, Moving on to the last one, which is exemplar similarity. So say you have a test instance. You're trying to classify the phrase, I'm so excited. And you have exemplars in your dataset that you want to put in the prompt. You can retrieve similar exemplars to that test instance, put them into the prompt, and then the LLM has a better chance of accurately predicting it because it sees similar exemplars to it. And in this case, it is necessary to dynamically generate the prompt at inference time. But there's a lot of techniques that do this. However, there are other papers that suggest if you put diverse exemplars into the prompt, then you'll have improved results. But generally, I found similar exemplars do the job much better. And there, it's more like there's specific cases where diverse exemplars might help. That is the end of the 6. Cool.

So first of all, what's the difference between an example and an exemplar?

Good question.

Since we're into the nitty gritty of the terms and the definitions.

Yeah. So you can kind of think of it as like exemplar is a technical term for the example that you're showing in the prompt. A lot of people say example equivalently. Doesn't really matter for the most part. But I try to use the technical word because it is more specific to this format. And you could say something like, oh, I have a dataset of examples that I want to put into my prompt, and only when they are actually in the prompt do I call them exemplars. So it's kind of like exemplars, just with a more technical appropriate term. That's the, the way I like to think about it.

Okay, cool. Hey, we'll continue our interview in a moment after a word from our sponsors. So to recap, the more the better. Randomly order them so you're not creating a bias. I suppose you could also argue for an intentional ordering, right? If random is better than all the positives before all the negatives or whatever, then interweaving them. But I guess then too, you might have— you could have an issue of creating a tick-tock sort of pattern that it might also be biased to follow. So you got lots of little issues there. The random ordering seems best if you're going to do some sort of ensemble or majority rules sort of technique.

Then you could do multiple random orderings.

But if you had to choose between random ordering and intentional ordering, do you think you can improve on random with an intentional setup?

Usually not. Yeah, I'd say the cases that are quite limited and be more of an instance of getting lucky.

Interesting. Now, when you're measuring these, are we measuring these at a sort of aggregate statistical level?

How do you mean?

Well, this is sort of in this kind of random versus intentional ordering of the exemplars. If I'm doing one thing in ChatGPT, Intuitively, it feels like I should set up, use my best guess as opposed to take a random guess. But then if I'm using a sort of large N benchmark to evaluate my strategy, then I might be like, well, yeah, maybe random there is better because I haven't seen all the examples and my intentional design is inherently biased because of the few cases that I considered when making it. So I guess the question there is really like, to what degree some of these techniques depend on large N evaluation of the technique versus just like, I'm trying to do one thing right now, what's going to get me the best results? I don't know if there's any data that would allow us to resolve those.

Right. I will say that we report these 6 techniques based on other papers that study them directly. And so they generally have some empirical results where they run it on a dataset and say, oh, like the random ordering is better because our ordering was biased or something like that. We did not run studies of our own with these different pieces of advice, but they do correspond to my anecdotal experience, which is not limited. Really. And also my intuition as far as prompting goes.

Okay, cool. So again, just to quickly recap, the more exemplars, the better. Use random ordering, or at least we can very confidently say beware the creation of bias in your ordering. Have a distribution that is ideally representative. Obviously, look at your data. Probably the number one commandment in all of LLM usage, make sure that the exemplars that you're using are correct. I think the format one is really interesting. I wonder if you have a, you know, like, what's a good format? What's a common format? What's an intuitive format? One idea that comes to mind for me is like, just ask the language model to like format the thing in the most intuitive way for it. What would you say?

I almost always use the Q input, A output format. I've tried augmenting that To be like, instead of Q&A, question and answer, but Q&A seems to work really well. 'Cause, well, I suppose because historically that's been what has been used in a lot of datasets. And so LLMs have kind of been trained to know that format really well.

If you're doing some other task though, that's not like a strict Q&A, do you like the intuition? Oh, you still use it. I'm thinking about like the Waymark use case. Are job is to write a script for a video. Now, I suppose I could just put like Q colon everything that I have in the prompt and then A colon and then get the response. That feels odd, but I can't say it wouldn't work. What do you think about my intuition, though? I've developed my syntax for what is the structure of a prompt, kind of using my own intuition, trying to keep it minimalist, trying to keep it readable, trying to keep it clear. I will say, though, I didn't study the perplexity of every token. And so there could be some tokens in there that are placing more strain on the model than I would like. I guess looking at perplexity, I just said, would be one interesting way to kind of spot ways in which your prompt is making the model work too hard or harder than you want it to. And then my other idea is just giving it my thing and saying, reformat this in the most obvious way possible. Those feel like they are good ideas, or how would you riff on that?

Yeah, so let's see. I guess my general advice would be the best thing to do is use your domain expertise with an understanding of what is standard. And to give you an example of someone not following this advice, I was recently watching someone try to analyze stock data, and they were formatting their exemplars As like big paragraphs all put together. And they would put in like, if you see this input in quotes, comma, then output this. And then they would have like another one and be like, oh, however, if you see this input instead, then output this. And so it was a very inconsistent format where they would— it wasn't really very structured and they were just using English, well, of course English language, but they were formatting it as kind of an essay about what to and not to do. Whereas what I always do is have a structured input output, as I said, the QA, and that's what I've seen to be standard. And so someone like that could have benefited from knowing, oh, like if I do something like QA, that'll be a bit better than what I currently have. And so I'm sure what you're doing is fine, and I doubt that you would benefit that much from switching to the QA, for example.

How do you think about that in the context of elaborate instructions though? I mean, the examples in this figure are all relatively simple. They're basically classification tasks. But if I'm doing a generative task where I do have these more elaborate rules, and we do have this at Waymark where it's like, we want, for example, The, we're writing scripts for small businesses, right? So inevitably there's going to be like contact information in there, like visit us at our website or call us today or come to our store or whatever. And we want to use those in like the right place. And so we sort of have instructions that are like, you know, use the contact information where shown in the example. But sometimes the business itself has a brand that is contact information. Like in Michigan where I live, there's the legendary 1-800-CALL-SAM. And, you know, that's like the brand is the contact information. So we have like these kind of caveated exceptions. Is there any way to present that to the language model that would be more intuitive or, you know, more helpful to it? How do I play to its strengths when I have these sort of elaborate constructions?

When I have these tasks, I generally do, I keep a structured format. But like, I'll probably use something like instead of Q&A, I'll say like, this is the input colon and put the input there, put an enter or 2, and then say like, this is the output. And then after each exemplar, put like 10 equal signs or something like that, just to try to show the LLM that, okay, this thing was just one exemplar all by itself. Because I don't want it to like kind of merge multiple exemplars together and I think they're all one thing. That's what my intuition has told me. I would love to see a paper come out analyzing all these different formats across models. And this would actually be a decently easy paper to write because you just have to do a bunch of different runs through the models with a number of different formats. But until then, I can't really say anything besides my intuition there.

Okay, cool. Well, if you're listening and you want to volunteer as tribute for this project, reach out to me and/or Sandra. We'll see see if we can't give you some direction. More you than me. I'm volunteering you mostly there. I'll chip in, but I think you've got deeper expertise than I do.

Absolutely.

So the final one, just to finish my recap, is exemplar similarity. This is like kind of a RAG style, if I understand correctly, strategy where it's like we've got a big database of examples and then we want to pull in some. And I would also imagine probably doing this maybe in like a hybrid way where I maybe would like Fix a couple of them and then add in a couple more dynamically at runtime to try to make sure I have some control still, but that I'm also finding the most similar ones to give it the best guidance that I possibly can. That's good. I think that's really practical stuff that people can take and run with. How much do you see this varying across models? I mean, I don't know how well you sleep at night, but if you're up worrying about the generalization of your research, I would imagine one big worry would be like, shit, how does this apply to Gemini 1.5, you know, next or whatever, right? How much variation do you see?

This is honestly a kind of a question that I hate, not because it's a bad question, but because it's such a good question and I can't answer it well. In the paper, we did studies with really just one model, which was GPT-3.5. I would have loved to do it across a lot of different models. In fact, we even Did it to some extent with GPT-4, but we found that time more than anything was restricting our ability to do all this. And not time in terms of setup time, in terms of running all these prompts. It would've taken like about a month to run all of this through the models we would've wanted, which would've included GPT-4 and others. And that's because we're doing it through these sort of public APIs, which are slower than training API, or if you have access to the base model like maybe Scale does, you can run these evals a lot faster. So I am concerned about how these techniques do generalize, but I think the few-shot advice is sound across models. And also every technique in that paper can be implemented with any standard LLM. So OpenAI, Claude, Google, all of those can be used to implement these techniques. There's probably some performance difference across them as far as reasoning and ability to break down problems goes, but I'm only reasonably concerned, not like too concerned, I guess.

Okay. Sounds good. How about on the, another sort of weirdness, right? We see these examples all the time where some like seemingly innocuous change to a prompt can really make the difference between good and bad performance. Some of that may just be noise, but even at temperature zero, it seems like we've got examples where I add one extra space between sentences or whatever, and it's like, oh my God, something seemingly so inconsequential becomes quite consequential in some cases. Do you have any sense of how real of a problem that is? I mean, I don't want to be dwelling too much on just cherry-picked random examples, but I honestly have no idea if that's something that I really should be Worried about. I've always been a 2-space person between sentences, and I've stood up for that for a long time. However, if it's going to hurt my language model performance, then that would be like one really good reason to reconsider.

Interesting. I am definitely a 1-space person. So what you're saying, it is absolutely a problem. I think we talk about it in our like reliability or alignment section. It should be less of a problem over time as the models get better. And in fact, I Believe that like GPT-4 is much, much better at dealing with that than like GPT-3 was. I honestly haven't seen anything recently about this having a big impact on prompt performance. And whenever I'm developing a prompt, I'm not like studying, oh, if I change this word or remove this one word, is that gonna have a big impact? So it's kind of something that I just know is there, but can't do too much about and generally assume won't affect performance too much.

Okay, cool. I guess basically the trend is that it's less of a problem over time. And it seems like with kind of maybe more epochs of training, more overfitting in general, and more RLHF, we're just kind of gradually hammering these oddities out of the models. Is that basically the underlying story?

I would say more and better training. Honestly, I'm not sure about the RLHF part, because I think that has been shown to decrease accuracy and so maybe would introduce more of those oddities. I think RLHF in general decreases the accuracy because it's kind of like a trade-off between model accuracy and niceness of human interaction.

Interesting. So you think you can get better performance? I certainly have experienced this on some tasks, like joke writing, for example, or imitating a certain famous writer's style, you get a way better imitation out of the base model than you do out of the RLHF model. Also, just like random number generation has been reported as way better from base models as opposed to RLHF. You think that's like an across-the-board trade-off? If it was something like a coding task, I would have assumed that the RLHF models would give you more accurate code than the base models would have. But I don't know, maybe not.

I think that really depends on how you're using the RLHF. If the learning signal is like code accuracy or how much a human likes the code and likes how it looks like, the latter is a poorer signal of performance than the former, which is directly based off of code performance.

Yeah. Okay. That makes sense.

And I guess as far as like the sort of RLHF safety tuning goes, I've been reasonably frustrated with GPT-4 because it's like too closed off. So I've been trying to understand credit scores, like my credit score recently better, and trying to get the model to tell me, you know, how the credit score calculation might work. And it's like, oh, it's proprietary information. So I'm like, oh, well, Could you just give me like an example of how the formula might look? And it says, oh, like it might be harmful to output such information. And, you know, I'm trying to learn and that's pretty frustrating. So, you know, that's, I don't want to see that.

So do you use base models regularly? I was just looking at Llama 3 70B and comparing instruct versus not instruct on the Together AI. API, they do have both the instruct and non-instruct set up there for people to play with and to call via API. Is that like, how often do you go to a base model?

No, no. I generally only use ChatGPT for Claude for coding tasks. Those are my 2 mains at the moment. I don't use a lot of base models unless I'm going to Do a research paper using them.

Interesting. Well, you might actually have a pretty good experience with that. I was trying, looking at a couple different things like synthetic opinion data and trying to think about how could we kind of take a seed of somebody's opinion profile and expand on it. And my intuition is that the base models would probably be better for that. And then also with this impersonation task, it's like, or again, joke writing, it's like, can be a lot better. The RLHF models are not funny. I would say that's like, they're pretty limited in that regard. Even just titling the podcast, sometimes I have, like, I know what I want to title it. Other times I'm like, can I get something a little punny? I need to start going to the base models a little bit more because the RLHF ones are just so often a little boring. And I think I can maybe get some more outside the box ideas. Probably just higher variance in general. I'm sure if you were to score them, take the average score across all the ideas generated, they would probably score worse. But where I'm really looking for the highest max score, then I think you would find something often from the base models that could be better. Okay, cool. Let me just pitch you a notion of how I typically work and how I advise others to work, and tell me if you would change it at all. Obviously you've got tremendous detail here, but if somebody comes to me and they're like, hey, I'm trying to get an AI to do X task and it's kind of complicated. I asked it to do it. It's not really working, or it's not working well enough. What do I do next? What I always tell them is staple your pants to the chair for a couple of hours and write out a handful of gold standard examples. Where you're going to explain your reasoning in painful detail on your way toward the actual output that you want. And most of these outputs, obviously, with benchmarks, it's very helpful to have sort of a ground truth answer. But a lot of what people actually want to do in applications and in workflows in their businesses and whatnot is not limited to a classification or a multiple choice where there's like a concrete ground truth right answer. People want it to generate something and they want that thing to be good. So I'm always like, kind of use the task decomposition, really think super hard, introspect hard on what are the logical jumps that you're making, what are the cues that you're zeroing in on that really matter. Try to get explicit about that stuff to the maximum degree possible. When you think you've said it all, go back and try to be even more explicit about all the hints, all the cues, et cetera, that you're using. And then finally work your way up to generating something good. And then with a handful of these, you are probably going to get much better results. If that still isn't good enough, then maybe you could think about using that as like a bootstrap into a fine-tuning loop, which I know is out of scope for this paper. But how does that compare to the advice that you give people who are, you know, not 76 pages into all the different techniques, but are like, this isn't working for me. What do I do? You know, short answer.

Honestly, I think I'd have pretty much the same advice. The exemplars are super important. My favorite way to improve the prompt, because a lot of people come in and, you know, just put their instructions and hope that it'll do the job, but Showing multiple, at least 3 examples of what should be done can really help. So completely agree with you there.

Okay, cool. Good to know. Next section, I have just kind of a number of questions that came to mind as I was reading through the paper that are on kind of specific subdomains. On ensembling, there's a lot of different ensembling techniques, of course. Mostly, again, they seem to sort of be predicated on this ground truth or some sort of discrete nature of the answer. Where if it's a classification task or multiple choice, if you run a model 10 times and you take the most common answer, that'll help boost your overall correctness score. Is there any way to extend to domains that are not like single answer that's like right or wrong, but a more kind of open-ended generative task? I would love something there if there's any known techniques.

Okay. Do you have ground truth examples of what you like?

Yeah, I'll say yes. Or I can make them.

Maybe I missed part of the question. Would you mind reiterating the question?

Yeah, I guess a somewhat abstracted way to say this is we know that there are a number of ways to spend more compute to get better results. And you can do that by chain of thought. It's often described that way. You can do it by this sort of majority Vote if you have something that's concrete to vote on. But if I want to write a video script for Waymark, for example, or anything along those lines where it's like there's not a right answer, but there are definitely better and worse responses, how can I spend more compute to get a better video generation? Maybe it's not ensembling, maybe it's like self-criticism. That was going to be my next question. But just beyond the simple chain of thought, how do I Spend more tokens, spend more money to get better results.

Right. All right. I do understand your question now. Thank you for clarifying. I do have an answer, but it's kind of frustrating in that it's a lot more complicated than for, you know, techniques where you do have simple accuracy-based metrics. And so number one, that would be like you have some pipeline where you generate the initial script and then you pass it With a, like a new prompt that says, oh, you know, revise this script according to insert blank guidelines that you want. And then maybe you even have like a couple different parts of the script generated independently. So you have your outline, maybe then there's a criticism of the outline, and then the new outline gets passed in, say, you know, generate this script, and then you have the bot criticize each part individually. Another thing you could do is a multi-agent setup where you give each agent a role where one is the sort of main writer and then there's a grammar checker and a style checker and they all kind of work together. I've seen a couple papers coming out using that multi-agent technique and even multi-agent debate techniques, which are very interesting. I don't think they're there yet or it'd be worth spending time implementing them. But the former, where you have sort of the self-criticism according to some guidelines, might be worthwhile. But if I were doing this, I probably wouldn't implement it because I still feel like the techniques are not there yet where it would be super helpful. It's a bit of a frustrating domain where I don't know of a good solution to the problem.

So maybe the better way to spend compute for those kinds of things is going back to that, like, bootstrapping into fine-tuning loop. The second part of my advice to people is if you've written, let's say, at least 5, maybe 10 really detailed, painstaking gold standard examples, and that's still not working for you, then I say, okay, now start generating. First of all, you could do just more generation by hand, but people typically tire of that pretty quickly, especially because they're going to need another order of magnitude in all likelihood. So my next advice is beyond pure prompting, but use those examples with the best available model to generate more examples. Now you have a human classification problem of which you like and which you don't like. Basically expand your dataset of things that you like by generating and filtering, and then fine-tune on those. And hopefully with 10x more, I usually think of it as start with a couple examples, fine-tune potentially with as few as 10. Certainly fine-tune again at 100, maybe go to 1,000. Hopefully you'll both get more robust coverage. Hopefully you'll get better performance, and you're just spending that compute at the fine-tuning stage instead of the inference time. Is that what you would recommend? I mean, it's fine for the purpose of whatever gets best results. We're not limited to pure prompting. So is that where you would kind of steer people?

I like that idea in the same regard as I like the other 2 ideas I mentioned. It can definitely help, but I would not necessarily have more certainty in that technique than the other 2 prompting-based ones. But if it's working for you, is it working for you?

Yes. I mean, we could always debate, you know, how well it's working. Obviously we could always hope that it could work better. But I would say in our experience, yes, we have the biggest leap for us. With the Waymark task specifically was fine-tuning on chain-of-thought reasoning as opposed to fine-tuning purely on inputs and outputs. And then, yeah, it seems like a bit bigger dataset. We also kind of try to cover edge cases progressively over time. When we see things that aren't working, we try to go back and generate a few examples of those and include those in the fine-tuning dataset next time. And Yeah, I would say that broadly seems to be working. Cost-wise, it ends up being prohibitive, and latency-wise, to some degree, it ends up being prohibitive to try to cover all the edge cases in the runtime exemplars. The fine-tuning, at least with OpenAI, is more expensive, of course, but it is not as much more expensive as trying to put 20 examples into a single prompt to cover everything. So I would say it is working, although this also brings up the evals question, which is like, how do we really know and who are we trusting to make that decision? I wouldn't say we've entirely left the vibes domain in terms of how we're assessing this. We do have some structured evals, but we see enough edge cases there that we don't fully trust that. And we're definitely still trying to live by the look at the data and actually use the product regularly. Sort of approach, because I don't think there's anything automated that we trust to really tell us for sure if the new fine tune is better than the old fine tune or whatever. That's not the kind of thing I would say we can resolve through a purely automated approach as of now.

Right. Yeah, fair enough. That is a pretty awesome development for y'all.

Okay, I've got like 5 more things, and I'll just limit my Side of the discussion. So multilingual is another area that you cover. I've seen in the past that English seems to be best. That seems to be the main takeaway. And so the kind of techniques here are like, translate in English first, then do the hard part, then translate back. I also noted an interesting bit from the paper on running certain tasks in multiple languages and then sort of a multilingual ensembling approach, which I thought was—

That's cool.

Very creative. I'd never seen that before. Any other notes or surprises, fun tidbits from the multilingual?

Actually, I was going to say the second one that you said, because that was my favorite as well.

Okay, cool. How about multimodal? This is, I feel like, very much kind of black magic. I've seen a number of things that seem like they help, but it seems like it's just all way less studied, way less established. And that's kind of a reflection of where we are in AI generally, where the capabilities are coming online faster than they can be systematically studied. But what jumps out to you from multimodal?

Yeah, so my favorite one there is chain of image prompting, where the GenAI has some sort of visual reasoning question like, given a line y= x and a circle defined in its sort of mathematical function, Do they cross? And so what it can do is, okay, it graphs the circle and then graphs the line and then generates an image, renders an image using maybe like some SVG Python package, looks at the image, and then prompts itself to ask itself, do they cross? And then there's like sort of longer chains of reasoning you can do there. So I really like that one because it was a direct adaptation of chain of thought. to the image domain using images as the steps in the thought process. So I like that a lot, but kind of bigger picture, Gen 3 came out with Runway that I've been experimenting with that, and I found that prompting it is extraordinarily difficult. So where we were with like, I don't know, GPT-2 prompting, GPT-3 prompting at the very beginning where you had to be really, really specific about a lot of things. And, you know, similar with the image prompting things. With video, it's even worse. You need to be like a really good prompt engineer in order to get decent results in video. Though I haven't gotten to try Sora, so maybe something like that or Kling is a lot easier. But Gen 3 for me, and even one of my friends who is more specifically focused on video prompt engineering, quite difficult to get the result you want.

Yeah, those are dense. The examples that I've seen are super. And just to make clear here, we're kind of crossing over between multimodal prompting of your sort of like large multimodal models, like your GPT-4Os. And then you're also kind of talking into the asset generation side, and that's a whole other beast that I honestly haven't studied super deeply. But the density of the prompts that I have seen in Runway 3 so far is pretty intense. You're talking like whole paragraphs to generate a few seconds worth of video, and people are being very precise in terms of the compositions that they want and the transitions that they want. I think DALL-E 3 basically has that same kind of super dense prompt, although they in ChatGPT kind of generate that for you and extrapolate your presumably generally naive— not you, but the royal you, the common user's naive prompt. They kind of extrapolate that into a very dense prompt and then give you the results of that back, and you can sort of iterate on that.

Right.

I think in general, my sense of the asset generation things is that the trend is toward very dense, prompting, where it's again just being extremely explicit and painstakingly detailed in what you want, is kind of what they are rewarding now, even if they're sort of abstracting that out of the user experience in some cases. I've seen a couple other things in image, going back to the image inputs to language models, where overlaying a grid or sort of overlaying the output of an image segmentation model, Or in some cases, like running a dedicated OCR model and then taking those outputs and including that with the image can be really helpful. I've been trying to do some things with looking at plumbing parts catalogues, of all things, which in many cases were originally paper and then they were scanned. And so there's kind of fidelity issues on top of the fact that these were definitely not intended for language models, but the OCR seemed to really help there. If we just ask any of the frontier models to do the full thing without the benefit of the pre-OCR processing, it can't get the parts numbers right. But if you give it those parts numbers in text and then also the diagram, you get a much better output that actually returns the appropriate parts numbers.

Very interesting.

Does that prompt bring anything else to mind for you?

No, not in particular. I have kept my prompting mostly in the text-only domain until recently experimenting with content generation stuff. Yeah, cool.

We'll compare notes about that another time. Agents, obviously super hot topic. Conventional wisdom is they don't quite work. We did an episode once with one of the founders of Codium. They put out a paper called Flow Engineering, which I really liked, which we exchanged some notes about. Any hidden gems in the study of agent prompting?

I agree with the conventional wisdom that agents don't quite work. I like the simpler approaches, something like React, where you, you have a super simple interaction with an environment. And I think you can get tool use working reliably well when it's very specific tools. Overall, I am very excited about agents, but I think this is gonna be something where we kind of need a next step in Architecture to make tool use more of a first-order subject. Because right now it's like, oh, you prompt the, or you fine-tune the LLM to output some specific wording to use the tool, but you could also prompt it to output a new type of token. Or I really love reinforcement learning. I think that LLMs or whatever comes after LLMs are gonna have to learn on their own. And produce their own reward signal. And so I think RL is the way to do that. Hopefully eventually RL will have its like NLP LM moment. Agents right now are quite frustrating because it looks so promising, but in practice it's so hard to get stuff working, even for me. So do I have any like wonderful takeaways here? Honestly, not really. There's a ton of different approaches to agents. They can do specific things somewhat well. But, you know, why can't I have something that I say, oh, I just got an email, I need to pay someone, go to my bank website and send them this much money. You know, that seems pretty straightforward. I could just say that and it strings a couple website URLs together, but, you know, I can't have that right now and I want that.

Sounds like a fast way to drain your bank account at the moment. True.

And security concerns are, Huge. Cool.

I think actually, to some degree, I've buried one of the leads here, which is the really interesting finding that after all of your intensive study, you were brave enough to face off head to head with an automated system for prompt optimisation. Specifically, I actually don't know how this is normally verbalised, but I think it's DSPy. Maybe it's verbalised another way. This is sort of a weird project that A lot of people rave about that I've kind of studied a little bit and can't say I totally get it, or it doesn't feel super intuitive to me, but it's sort of an optimization framework that can kind of optimize everything in your system, including the prompt. And I guess at least under certain controlled conditions, it beat you on a head-to-head. So tell me that story. Give me a little more intuition for this. thing and tell me what that means for the future of prompt engineering as a profession.

Sure. Let me start with the naming. I call it DisPy, and I think a lot of the people in my lab do as well. But DisPy, or I guess DSPy, also seem valid. Actually, Omar came and talked at UMD, and I was there for his talk, but I don't remember exactly how he pronounced it, unfortunately. But onto the prompting stuff. Yeah, so I spent like 20 hours developing this prompt for a binary classification task, and I was pretty happy with it. And then my advisor used Dispy and put in like the training data that I used as exemplars, and Dispy was able to create a prompt with the exact same data. That I had that blew me out of the water on the test set. So as far as does automated prompt engineering work, uh, I, I can now say yes, it does. And I think you do need like ground truth examples. I don't think it can optimize just any prompt because it needs some kind of signal to optimize towards, but I was super, super impressed with it. I definitely didn't think it could. Yeah, I didn't think I could be defeated by an automated prompt engineering system.

And this thing can use proprietary models, right? It doesn't, like, black box. It doesn't have to have access to the weights.

That's correct. That is correct.

Yeah, I was pretty sure about that. How much data did you have and how much data did it also have?

Let's see. Probably gave it less than 20 exemplars to use and we had Maybe a couple hundred. The dataset itself was quite small.

Interesting. One wonders if it could be used to help solve the ARC challenge. That's connecting a couple of threads. Cool. Well, down to the last couple of minutes here. Any thoughts on productizing all this? I mean, obviously you've got the Learn Prompting. I can sort of imagine a lot of businesses would be interested in an extension of Learn Prompting. That is like, I mean, you have content, you have instruction, you have best practices, but what about a prompt coach that follows me around and applies this taxonomy to what I'm doing in my day-to-day use of AI and coaches me on how to get better? I'm sure that's something you've at least thought about. And then with the DSPY or DSPY, whatever sort of results suggest that you wouldn't even necessarily be leaving much on the table. In terms of theoretical performance, to follow the advice of a system. Claude has done that. Anthropic has done that a little bit with Claude. Anything in the works there, or are we going to leave that to others?

Right now we do have a RAG chatbot you can ask questions on about our docs. But as far as putting in your prompt and getting advice on it, I could see it being useful. I think it is quite a struggle because in different domains, people are just doing things that look completely different. And so I'm not sure if we would be able to prompt the model well to do consistently well across domains. And then I also don't know if that's something people would be willing to pay for, like, at all. But it is something we considered, not a main product we're focusing on at the moment. But if we saw interest, we would look more closely in that direction. Cool.

Okay. Well, it's a tour de force paper with a ton of— for me, I would say now it is kind of the starting point for any inquiry into available prompting techniques and a great place to navigate your way toward deeper studies of all sorts of different things. Anything else that we didn't cover that you think bears mention or highlight before we break?

Yes, that would be sort of my takeaways from running a team of this size. And let's see, I'm just, I have my like reflection document. I did a, what I later learned is called a 360 review at the end of the project where I wrote a reflection about my performance, then had my team do that anonymously as well. And so one of my conclusions was trust no one, not even yourself. And in practice, that means something like you always have tests, you always have a CI pipeline. So anytime someone makes a PR, including me, All those tests and styling checks run against it. And so you make sure you don't have any regressions. I found that to be super helpful in past projects and no different on this one. So overall, I don't recommend doing a systematic literature review because this took me twice as long. You know, I ended up taking like 9 months, but I, I did enjoy doing it. I am happy I did it and I really learned. Quite a lot doing so.

Well, your hard work is to the community's benefit and to our benefit today. So thank you for putting in all that blood, sweat, and tears, elbow grease and everything else, and organizational legwork as well. I think it is a great resource and I appreciate you coming on and returning to talk it through with us. I know you gotta go. So for now, Xander Shulhof, thank you again for being part of the Cognitive Revolution.

Thank you very much.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.