From Poetry to Programming: The Evolution of Prompt Engineering with Riley Goodside of Scale AI

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years.

I'm Nathan Labenz, joined by my co-host, Erik Thornberg. Hello and welcome back to The Cognitive Revolution. This is Nathan's AI voice clone powered by ElevenLabs. The real Nathan is in Brazil this week to give a presentation on AI automation at the ADAPTA Summit in São Paulo. And today's episode, featuring returning guest Riley Goodside, the world's first staff prompt engineer at Scale AI, turned out to be a perfect prelude to that presentation, which we'll share here soon as well. Going back to 2022 and into early 2023, Riley became famous in the AI community for coming up with one clever prompting trick after another. He was one of the first to put large language models into a read, evaluate, print loop or REPL, which most of us would now recognize as a precursor to AI agents. And I think you could make a strong case that he had demonstrated the deepest, most practically useful understanding of language models' idiosyncrasies and outright weirdness of anyone in the world. Today, our conversation reflects how much LLMs have continued to progress. Even since GPT-4. As models have undergone more and more post-training, the need for quirky tricks has declined. And as Riley puts it, prompt engineering has become less like poetry and more like programming. Meanwhile, in the enterprise context that Scale AI serves, companies are starting to move beyond ad hoc chatbot interactions and instead using the full range of best practices, including curating gold standard examples. Capturing reasoning traces, all sorts of retrieval, augmented generation, fine-tuning, and anything else they can come up with. All to push language models to their performance limits, with the goal of achieving human-level performance or better, and ultimately saving serious time and money on routine tasks. It was a treat for me to dig into these topics with Riley, and I was glad to find that our approaches are mostly in sync. For anyone building AI systems, This episode is packed with practical value. And of course, I couldn't help myself from sneaking in some questions about jailbreaking, the prospect for superhuman intelligence, AI safety, and more along the way as well. As always, if you're finding value in the show, we'd love it if you'd share it with a friend. Your support helps us continue bringing you conversations with the leading minds in AI. The real Nathan will be back soon, but for now, he hopes you enjoy this deep dive into the evolution of prompt engineering. With Riley Goodside of Scale AI.

Riley Goodside, the world's first staff prompt engineer of Scale AI. Welcome back to The Cognitive Revolution.

Thank you.

It's great to be back.

It's been a minute and obviously a lot has happened. In preparing for this, I went back to your Twitter feed, which was of course your original claim to fame in the AI space with lots of super interesting examples coming out of the Text-DaVinci-002 era. And I found you've been a little quiet on Twitter recently. I think the AI community has certainly grown. There's lots of people doing that kind of stuff these days, but what have you been up to that's had you quiet online in recent months?

Yeah, so there's a few reasons I've dropped off in my posting activity a bit. So the biggest reason I've been quieter on Twitter is that I'm a father now. We, June of 2023, We welcomed our first child, our daughter Felicity, into the world. And it's been magical, but of course also time-consuming. So that's between that and working, it's left a bit less time for extracurricular posting on Twitter.

So not the total lack of progress since GPT-4 was introduced that has—

No, no.

Just given you nothing to talk about.

No, I think it's, you know, certainly not a lack of progress. I think that there are other subtle More subtle factors at play, though. So one, I think the other thing that's maybe a more mundane reason is just changes to my situation personally at work. I think to some extent my activity on Twitter was a campaign to break into the AI industry and get me hired. It did that job. So I have a little less motivation. Also, when you work for a company whose customers include much of the AI industry, you have more people that you have to be sensitive towards. So it's harder to do some of the more irreverent material, I guess, I was known for earlier on. And I think the other thing too is really just that I think maybe dominates in some ways are changes to prompt engineering itself. I think this is maybe like a bigger topic that we could get into later in the podcast, but I guess my TL;DR teaser for it would be that I think modern prompt engineering is becoming more programming than poetry. And nobody really wants to do code review on x.com. We'll come back to that, but I'll leave it at that for now.

Okay, cool. Yeah, that's interesting. Let's get to that very soon. I think my comment on lack of progress since GPT-4 was hopefully understood as a joke. It was meant to sort of allude to the sort of commentariat that's, oh, we've hit a wall. Nothing new has happened since GPT-4.0 or GPT-4. It's, you know, it's all stalled out. That's definitely not my perspective, as somebody who is an intensive user and increasingly seeing customers of scale, presumably using language models in a lot of different contexts, what stands out to you most in terms of the new capabilities that have come online? What do you think has really been the biggest difference maker or driven the most value over the last, say, year and a third since we first got to see GPT-4 in public?

I think a lot of the progress lately It's becoming maybe harder to describe, maybe a bit more diffuse across many different tasks. One area of capability that I've been excited about for a long time really is use of REPLs and the ChatGPT code interpreter, as it used to be known. I think it's now Advanced Data Analysis and other projects that extend this idea even further, like Julius AI, of giving the AI like an interpreter environment to work in. I think there's a lot of progress that's been made and still a lot of alpha to be found in increases to that capability, right? We're in this regime now, I think, where LLMs could benefit a lot from knowing how to use LLMs, right? That there are still things to know as a user about what tasks an LLM is good for. It's not quite as much as it used to be. For example, like GPT-3 would, would very reliably not, never say, I don't know. If you asked it a question that had no answer, it would still just make one up and you had to remind it that, that, that saying, I don't know, is an option. Right. And that sort of knowledge is, it's becoming easier, but it's still there. Right. Like the things that you have to know to understand that this is a classic example of a tokenization problem that an LLM would struggle with. LLMs don't have that sort of, I don't know if you would call it situational awareness, but I guess it's a form of that, of advanced situational awareness of the circumstance that they themselves are in as LLMs. So I think that there's still a lot to be gained from just these kind of like bread and butter improvements of training on data that reflects usage of these features. Right. I think in some sense, like when the internet was scraped to train, say, GPT-3, it was an internet that didn't contain examples of chatbot dialogues as we see all the time today. Right. So even just, you think of in pre-training. You learn more about how a chatbot behaves scraping from the modern internet than you did before. So there was more that had to be built in post-training in order to flesh out the sort of fictional character of how a chatbot stereotypically behaves. That's very exciting to me, that more of this data is disseminating through the culture, getting filtered. People are picking out which dialogues are worth sharing and worth remembering and high quality. And the best parts of this are going to be fed back into the models and they're going to have more of this very practical skill that, that I think they're uniquely lacking in and uniquely in need of being able to use themselves and be able to use other LLMs and understand like the limits of their capabilities. So I think code is, I think, the most promising avenue for that. In code, you can represent understanding of a very wide variety of systems. And LLMs are like one such system that you could have code that uses LLMs. that code can be understood. And I think that we're going to see a lot of lift just from that sort of recursive process continuing.

So that sort of obviously suggests a focus on agents, to put it plainly. I guess 2 questions I have there are, part one is, how would you characterise the frontier of capabilities right now when it comes to tool use? And you could break that down in terms of how many tools can a language model be provided and still work well, or how complex the Maybe some tips for, or do we need few-shot to get them to work well with tools? I think a lot of people have seen this stuff, but I think very few people still have much experience in terms of actually getting deeply hands-on with tool use.

I think for tool use, it's often the case where you want to have fine-tuning just because there's a lot of subtleties that go into using real-world APIs that you would need to communicate to the model. That can be done through long context. It can be done through examples. And that's often a good way to get the data that you would want to tune on. But I'd say that's a case where you're more likely than other tasks to see like a need for fine-tuning to get like the reliability that you would want for something that's going to call out to other APIs. You can otherwise see issues with it forgetting like just details of the semantics of like how some JSON structure is to be called or so on. In cases like that, where you have a very rigid structure of your generation, it doesn't take many examples to help.

Maybe to drill in a little bit on fine-tuning of language models for this tool use capability, how general versus broad do you think people should be thinking of? Most of the fine-tuning I've done has been single task, and that makes my life pretty simple in the sense that I can just focus on the single task. I'm trying to maximise performance. I'm controlling the environment in such a way where this model's never going to be asked to do anything besides this task. And so that makes the situation relatively easy from an evaluation standpoint. If you're in a tool use domain, maybe you can narrow it down to that super specific task where you're like, it just has to do the one thing. But inherent in tool use is that you want a little bit more dynamism out of the system, or at least that's where my intuition is at the moment. So then you have the challenge of, man, I'm starting with a base model that a world-leading company has created to be like a very general purpose tool, and I'm going to narrow it consciously or unconsciously through this fine tuning process, but I still want it to be somewhat general. How do you think about managing, first of all, how broad or narrow you should aim for? And then how do you actually keep things as robustly general as possible while you're doing Fine-tuning?

Yes. So the progression I usually use is you start with a prompt, right? Just a sort of instruction prompt or MVP of how you'd want the task done. Continue on to including K-shot examples, add more K-shot examples until no more will fit, and then start worrying about K-shot selection. That doesn't even necessarily have to use embeddings. Just pick some method of selecting K-shots and find something that works. And from there, there's a lot of extensions into RAG and so on. But only once you've maxed out those options, I'd say you want to dive into fine tuning. Both because they're, they're easier and lower investment, but also because it's not wasted effort. I think you get from the prompts and the especially like expensive long prompts that have wasteful amounts of context in them, you can get high quality answers that would be useful in say K-shot selection or fine tuning. So it's, you're not like losing anything through that work. But I think. The thing that people forget throughout this whole process of refining from like a prompt to a, like a more advanced prompt-engineered pipeline is decomposition. That at each step you really just want to be asking yourself, is there any piece of this task that I could pull out and make it easier and make it simpler? That I could delegate to a smaller LLM, that you could maybe delegate to something that isn't an LLM at all. That can really help a lot with performance, especially just digging into your data. And I think using your intuition of like, how could you fix this? Like, how could you fix this one example? Yeah. And make your data better just by tweaking your data a bit and providing more context and generating more examples to tune on. I think people really just neglect annotation really and labeling. They neglect just the need to put thought into coming up with examples of the problem, both in the sense of the doing the task well and having like high quality examples of their task being done, but also in correctly thinking about the distribution of inputs. Like what is a realistic input? to this task, what needs to be demonstrated, and also what are the edge cases? I think that's something that isn't emphasized enough, that you really want to be considering what are the boundary conditions through which your high-level task could break, right? Or where the output might be ambiguous or unclear to what you're looking for. And you want to define those cases so that it can see every real-world example as sort of an interpolation of the examples that you gave it. So I think there's a lot of roads you can take that lead to fine-tuning. But they all converge on that last step. You could have many different ways that you could produce high-quality data to tune on, but they all collapse into the same approach at the end. If you get to that point, right? And you may not, right? There are many cases where simply prompt engineering is the answer, and sometimes it's SOTA if you're willing to spend enough on it. I think there was a great instance of this. I think this was at Microsoft, built a prompt engineering pipeline. That managed to beat, I think it was Med-PaLM, that was like a version of PaLM that was fine-tuned for medical questions, just by using a non-fine-tuned, a non-specialist model that had a very complicated pipeline of prompt engineering on top of it.

Right.

And I think this is in some ways not always the answer you want to hear, that it's going to be complicated. That you're going to have this massive inference pipeline at the end. But for real-world problems, that's often how it ends up, that simply having more examples will get you there faster. I think one good intuition builder for this is that at the limits of K-shot selection, the problem is always trivial. So if you have enough examples that you can find this problem that you need to solve, then it just copies the answer. Right. So you can think of like K-shot selection as a sort of precomputation, and the space of what you could precompute is very wide open. And I think the computational approach of it, I think, is worthwhile, of thinking about what aspects of the reasoning and deduction that produces an answer are being performed at each step, and how can we split those out? How can we make them more explicit?

Hey, we'll continue our interview in a moment after a word from our sponsors. Mention of reasoning and mention of explicitness focuses my attention on the chain of thought or the reasoning traces. And by the way, I basically pursue essentially the same protocol that you mentioned. I'm actually working on a— I think it'll probably be an upcoming episode on just an introduction to AI automation best practices. And what you articulated there is pretty much what I'm trying to create flowcharts and whatever to distil for people. I have found personally that the quality of the reasoning traces in the examples is super important, and that is often an overlooked piece when people go and do this because it is often implicit. If you take, for example, the task of responding to a customer service ticket, what people will have is they'll have the ticket that came in and then they'll have the response that was sent. But the sort of internal notes, which usually does exist as a field in a ticket system, is often very minimally filled out, if at all. And it certainly does not contain the actual thought process that the person went through that's like, I'm going to do this because of this, and I noticed this, so that's got me thinking this. And getting that stuff out of people's heads and onto paper, so to speak, is something that takes some social engineering. Sometimes you got to go figure out who does this, how do they do it, even interview them to try to pull that information out. But I have found that to be a critical element to performance. I've done most of my fine tuning on GPT-3.5. We're now about to see, I think, a whole set of options of next generation models coming online for fine tuning as well. But at the 3.5 level, I found that if I just did inputs and outputs, it didn't seem to grok exactly what I was trying to get it to do. And it was really important to have that explicit reasoning trace there. So basically, yeah, I just wonder if you've seen the same or if you have any sort of refinements on that summary.

No, I think that's exactly right. That human-produced text, like if we think of like natural text that we find out in the wild, this text, sometimes it has sort of like one mode of behaviour. Where it resembles a sort of a recording of our conscious awareness of the thoughts in our heads, right? That you can see like reading something as being like playing moment by moment thoughts that a person was having as they were coming up with these words. But there are violations of that. Like a good one is calculation. Anytime you see in text something that says, just casually tells you what 57 times 125 is, you— are getting data that isn't part of that flow, right? It's the result of somebody stopping and pulling out a calculator and doing the math and then resuming the transcription of their thoughts after that. And that shows up in a lot of ways, that you see text that isn't always just what came to somebody's mind naturally as the next thing to say. It's the result of experience. It's the result of seeing something in the real world, possibly years of experience, history summarized or whatever. And that shows up in so many places and in so many tasks and so many things that we think of as like demonstrations of a task being performed. We're not really capturing all those pieces missing in the middle. And attempts to address this through the history of like attempts to address calculation errors in LLMs, right? So it's very apparent that LLMs struggle with calculation. And I think it's for kind of the reasons that I was saying, that in real life, like calculation summarizes over many steps. And we don't record those steps on paper. And if you let an LLM record those steps, if you let it think longhand and go through the steps of like reasoning through the digits, it's much more likely to be correct. Not perfect, but more likely. And getting that kind of thought out of your answers is very important. And I think it's also a classic case for the power of augmenting with synthetic data. Being able to generate like chain of thought rationales for particular answers. Especially, I think people don't appreciate always that there's a difference between chain of thought that the model generates itself and, say, chain of thought that somebody else generates. In that, if you were to imagine that you were to prompt a model on some— you have some dataset of questions for it to answer, and then for each one you ask it to think step by step and it generates some rationale for its answer. If you were to tune on those rationales, let's say you filter them to only the ones that lead to consensus answers or correct answers, but consensus being a proxy for that. You're tuning on rationales that empirically led to the correct answer for this particular model. That empirically, like when the model does this, when it zigs this way instead of zagging, it lands in the right place more often. And so it's going to learn these general reasoning steps that perhaps very idiosyncratically apply to this model about how it specifically should do reasoning. And that's not always true if you have just even like reasoning steps made by some other model or especially reasoning steps given by a human. That they're not really going to capture all of the deductions that they were making that led to things being decided. So I think the effort spent making that more explicit is rarely wasted, of breaking out your task into more substeps, annotating pieces of tasks. I think there's a lot of forms the data of a task demonstration can take if you put your mind to it. You can flip fields around, right? So every translation Every English-to-French translation is also an example of French-to-English translation. There's a lot of higher-order transformations you can do to your data to create a synthetic pipeline for producing the data that's worth tuning on. I think maybe like a more concrete example of this, it was a paper where they were trying to show that models behind fine-tuning APIs can be tuned to be unsafe. Even if the dataset that you're tuning on is monitored by another LLM to look for signs of unsafe activity. Because it's well known that as long as you can tune a model, you can give it something bad to— some examples of doing bad things, it will do those bad things. So the defense against this is that model providers will have filters that look at the data that you're tuning on and ask high-level questions like, what is the purpose of this? What are you attempting to achieve here with this fine-tune? And is that bad? So what they did to defeat this is they first came up with a text encoding method that they verified that this particular LLM couldn't understand. And I think that they called it Walnut53 or something like that. And it was just literally like you'd permute like ASCII characters by seed 53 in NumPy to get like a permutation order of them. And then you apply that as a cipher to the text. And they demonstrated that like the LLM without tuning could not understand cipher, even when it was explained and given examples and they did various tricks. But they found that they could get it to understand it through tuning. They created examples where a task is given in this new Walnut53 encoding, then the LLM's response is to repeat the task in plaintext, and then to complete the task in ciphertext in Walnut53. And after tuning on examples of this, I think that they had a few other tasks in there too that may have been like, like pure translation, things like that. But these were bridge tasks, just that their true purpose is just to get the model to the point that it could understand this particular encoding and then understand the payload, if you will, of task demonstrations that are, have the model doing unsafe things and are encoded with this cipher so that the filtering LLM can't understand them. I think this is a good case study in the sort of problem That an LLM just can't do before fine-tuning and how you would get it there, of breaking the task up into easier steps and incrementally getting it to be good at each of these smaller tasks and then composing them into the thing you actually want to achieve. So I think in a fine-tuning context, that can work as well. You can, like, multitask fine-tuning sometimes has its benefits. I mean, there's trade-offs there too, but I think just that decomposition is really one of the most important and overlooked parts of it, of saying, what are the pieces of this task that if it were good at, it could do better on the rest? Because even if your high-level task that you're trying to achieve is hard to demonstrate, if there's some particular skill that it's missing that isn't hard to demonstrate, it might be worth it to demonstrate it.

Interesting. Okay. That is a fascinating example. So just briefly on multitask fine-tuning, Let's say I do have one of these skills that a language model just can't do, whether it's like using my private APIs because that's not in the training set, or some weird cipher along the lines of what you just described. And I want to fine tune that in, but I also want to keep the generality, right? Like, my goal is to have an in-house chatbot where I would be able to say, hey, it's just like ChatGPT, or it's just like Claude, but it can use our own internal APIs. Sweet. Okay, cool. Now, if I want to do that fine tuning, I can't just give all the examples of— every example can't be using my internal APIs, right? Because then I'm going to presumably overfit and I'm going to start seeing the model start to want to use our in-house APIs when it shouldn't in actual day-to-day use, right? So is there any best practice recipe that you could point to that would be the sort of You handle your specific capabilities that you want to add to the model, and then you also mix in some other dataset of just like general chats or something. Like, what's the best practice for maintaining that generality while instilling the new thing that you want it to know?

Yeah, that's a great question. I think like the first piece of advice I would give is avoid the problem if you can. The fine-tuning does—

Wait for GPT-5.

That's one strategy. But more immediately, fine-tuning has a cost. It's not free. And one of those costs is that you lose more general skills. And there's also some more specific issues that show up. There was a paper not that long ago that showed that fine-tuning a model on any kind of new information, like new factual information, will make it more likely to hallucinate In general, like that, that it would be more likely to make up just information out of the blue. And logically, you can imagine why this is. The model has some understanding of what statements are true or come from this great body of text that we think of as the real world, or like describing the real world. And if you tune it on something that isn't in the that body of knowledge, you're telling it that new things are okay. The stuff that seemed weird to you in some deeply ingrained sense that you learned during pre-training isn't necessarily wrong. So ignore that instinct sometimes. And once you ignore enough of those instincts, you start to hallucinate. And I think that's like a good mental model to keep in mind of what you're losing is the opportunity cost for tuning. For what you can do. You mentioned a few strategies that are classic. One is mixing in more general data. The best thing you can do, of course, is if you somehow have access to the pre-trained data, but you don't. So you find some other similar source of chat data that you could mix in there. And that works well for preventing some of the forgetting. If you're dealing with like an open source model where you have full control of the weights, you can do like weight mixing. You can tune on your task and then evaluate on some benchmark of both your narrow fine-tuning task and some more general tasks that you want to preserve. And you would determine empirically what mixture of the weights optimizes the, both of those scores in the way that you care about. Another trick that I've heard anecdotally, and I'm not sure if there's a paper for this or if it's just something I heard offhand once on a Twitter thread, is when you fine-tune on your narrow task, if you have the option, say with like Llama 3, you have both a base model and an instruct model. What this thread was suggesting, and that it was saying that worked really well in practice, was tuning the base model rather than the instruct model, and then taking the delta of the weights between the tuned model and the base model and applying that delta to the instruction tuned model. This is a very crude outsider way of thinking about it. I'm probably wrong in the details, but my mental model of it is the fine tuning comes first, right? You think the process of RLHF. You tune it into a particular mode of instruction following before you put this finish of RL polish on it. And conceptually, you can almost think of it as a way of closer to sliding it in under the polish or not disturbing what was learned through RL. So I haven't actually tried that myself, but I've heard that it works.

Yeah, interesting. I think that sort of weight Frankensteining, which is my umbrella term for those sorts of techniques, That is super interesting, and I expect a lot more to come out of that, especially as we get deeper into the Llama 3 era and the Llama 3 400B era that allegedly will be beginning very shortly. It seems like there's probably a lot more that can still be figured out there in terms of how to just monkey around with weights in all kinds of different ways and still recombine them in useful ways. So that's a really interesting anecdote.

Hey, We'll continue our interview in a moment after a word from our sponsors.

What are your favourite models today and what are you most excited to fine-tune? Last time we did this, you gave us a great walkthrough of the stages of fine-tuning. You did Cho'Goth and the instruction tuning and the RLHF meme with explanation of how it worked and whatever. I wonder if you have a sort of characterisation of the different models today. What's OpenAI's series best for in your mind? Where does Claude shine? Where does Gemini shine? And how do you think that will potentially play out, especially as these fine-tuning things come online? Is there any heuristic that I should have for thinking about what to fine-tune first?

Yeah, that's a great question. I think the main thing I'd want to emphasise here is the best model is the one that does the best on your task. So whatever you're using the model for, you really want to evaluate all of the reasonable options because performance on different task domains is all over the map. You might find that one model is good at poetry, another one is good at code, and they don't have to be the same model. So it's worth testing. The leaderboards are a guide, but I think part of the strategy that we're really emphasizing with our SEAL leaderboards Is breaking out particular domains. So we have one leaderboard that's just for coding. We have another leaderboard that's precise instruction following. And we explain on the website the different procedures we go through for making these datasets. We have people coming up with role-play scenarios where you do an act-as-if exercise of imagining that you're in a certain professional role, coming up with the tasks you would do in that role. And then we have other people— Producing examples of those task demonstrations, so we get a diversity over the space of how a user would actually use the model. But I think having that domain specificity is really necessary even just for the initial guidance of what model to pick. And beyond that, I think asking around or even just looking at Local Llama on Reddit is a good place if you feel like smaller models and things like that. I think you'd be surprised at just how fiddly it can get. You would find that some models are much better than others at certain languages, so for if you need to work in a particular language, your ranking of which models are best could look very different. And also, I think in practice you might run into just annoying issues in one model or another. You might find that once it writes technical docs in Spanish, it sometimes just switches back to English. And then another model doesn't have that problem. And so that's something that maybe doesn't show up on benchmarks, but it matters a lot to you. So those are the kinds of things that I think People underestimate how frequent they are and how much and how little they show up in benchmarks. But I think you really want to use benchmarks as just like a guide to what to try and then to let data be your guide there. And also I think a lot of it is a cost exercise as much as anything that you really want to think through. What are the, your expected volumes and like the, how much are these things going to cost? Because like things can inflate fast, especially when you have like LN consensus pipelines and so on.

Is the difference in models today mostly something that is a result of different post-training strategies? I'm thinking here of this paper called the Platonic Representation Hypothesis, where I thought this was a really striking paper. Basically what they look at is the internal representations of concepts within different models at different scales. And they show that as the models get bigger, the representations appear to be converging. They become more similar as models get bigger. So in other words, your smaller models are more different. Your bigger models are more similar in terms of the internal representations that they have for concepts. Obviously, taken to the limit, you get something like the Platonic representation where it converges on reality and it's got some sort of compressed world model. Now, whether we ever actually get there or not, Maybe another interesting question I'd like to get your take on. But if that is happening, and if all of the big leading developers today are presumably tapping into largely similar giant pots of data, then one might infer that the reason there are different characterisation or different characters or different strengths and weaknesses, why is one better at code versus poetry or what have you, is potentially down to the post-training strategies and Maybe also explicit trade-offs that companies are making internally where they may be saying, we want to emphasise this over this for whatever reason. Is that how you would understand it?

Yeah, I think that's right. I think the longer I use LLMs, the more I see them as a sort of compression of text, a compression and an interpolation of the training data. And that sort of interpretation naturally applies to the perspective that these Models are getting better largely because of post-training, because of the human demonstrations that are going into them. And I think that's going to be increasingly true as we run out of room in scaling laws. We're not there yet. I'm not entirely qualified to say exactly how close we are, but my instinct is we still have a bit of runway there. But I think the broader trend though is clear that scaling will slow down at some point. And the gains beyond that are in post-training. But I don't see that as a major roadblock. I think that we don't even fully understand how much room there is to grow in post-training. I think the space of things that we could potentially be training models on is very wide open. I think maybe one of the clearest demonstrations of that is recent video models. We've seen that the same strategies that predict text can predict tokens that represent video, and this actually produces something that Which is realistic. So I think what we're going to find that there's a lot of things that can be turned into text. It wouldn't surprise me at all if a year from now you start having models that just produce executable binaries, right? Or other forms of data that, that, that like you wouldn't necessarily have to do a token representation of an image. It could just start spitting out JPEG. And maybe those are impractical or silly examples, or maybe like technical reasons for any given one of these things why it's not worth it. But I think in the general class, there are things like that are worth doing. Yeah.

Yeah, 100%. My mind immediately there jumps to biological sequence trained models and generations and DNA and RNA. Yeah. And those are still in the sort of GPT-2, maybe into the early GPT-3 kind of era. But those have a long way to go, I would think, as well. And that's going to get really weird, is my expectation, because nothing makes the, like, can a chatbot help me build a bomb feel more quaint than just the chatbot actually just straight away outputting a new virus sequence or whatever. And that is coming very soon, if not really already. It's maybe still not quite there in terms of actual viability, but scaling laws seem to hold there as well. So I guess most of the gains are coming from post-training. It seems pretty clear that all the gains over the last year and change have come from post-training, or the vast majority of them, because we haven't seen another order of magnitude scale up in model size. In fact, prices have dropped precipitously. So there's been presumably a lot of efficiency gains under the hood. I don't know how many parameters we're dealing with these days, but presumably the models we're calling now have fewer than the 175 billion parameters. So we haven't seen that scale up yet. I guess first question, just to be clear, you would expect that there is going to be another quantum leap with another scale up if we go from, say, 10 or 15 trillion tokens for GPT-4, Llama 3 to say 100 trillion tokens, you would expect that that would be like a qualitatively different model that gets spit out of the other end of that. Is that correct?

Yeah, there's sort of an irony in that we have these scaling laws that tell us how perplexed it will be by natural text, but how that translates into actual skills is really anybody's guess. But I think we certainly have only been surprised in one direction before.

Yeah. Okay, so now coming back to today's models, let's say you have time and resources to do the generations, to do the RAG pipeline, to do the fine-tuning, to iterate on it, to really put the elbow grease in to make the thing successful. Is there any guidance that you could give people on what is the limit of performance there? Another way to ask this would be like, how close are we to AGI? There's like a sort of state of fine-tunable AGI where it's like, If the definition is AI that can do the majority of work as well or better than a human, people tend to think of that as like, okay, just a single model doing it all. But if I were to reframe that as for any given task that humans do, can we fine tune? Can we find examples? Can we fine tune? Can we dial in the performance to get it there? How would you characterise the frontier of what is possible? Today, with all that extra work beyond the base models, I don't mean base as in not post-trained, the models that we're given by the companies and what we can get to if we do all that work that we've been talking about to really focus in on a specific thing. How could you characterise that frontier? Or are there things you could say, these are definitely out of bounds? Obviously, ARC has stimulated this conversation recently because the models are not that good at ARC. Even with a lot of examples, they're still not that great, maybe with fine-tuning, but it's unclear. Seems like it's probably not going to get to that 85%, even tuning GPT-4, although maybe, I don't know. If we were to do that elbow grease for every given task of interest, how close are we to that AGI definition is what I'm trying to get at.

I think the definition of AGI that's out there, which is probably heavily influenced, I think, by OpenAI's definition of systems that can do the vast majority of all economically valuable human labour or something like that. The issue I have with these kinds of definitions is that, like, on the highest level, this is a moving target, right? If you think of, like, what is economically valuable work that humans put out, before, like, the Industrial Revolution, most, probably, like, around, like, half of all humans were involved in farming. Somehow you were somewhere in the chain of, like, agriculture. And the number, I believe, is, like, now, like, 2% or something like that of all Americans are, like, would, like, work in farming. And so by that metric, that's 48% right there, right? Of like the life of human labor that got automated by tractors and the other farm equipment we've come up with. But like when things like that happen, it changes what it is that humans are doing, right? That we find other things to do with our time and economically valuable human labor becomes something else. And I think we are seeing like smaller versions of that even now with say translation. That LLMs are increasingly good at translation, transcription of videos. Like, those tasks are pushing humans out of that work and into other work. And we'll see, as humans get pushed out of work that's automated, they'll move into other tasks where LLMs, you know, aren't as good. And our definition of, like, what is, you know, human labor will change. Now, that's maybe a bit of a cop out because you could still say, okay, let's freeze it now at 2024 and say, like, of all these things, let's do those. But I think, like, even there, it doesn't get us to the thing that I think a lot of people implicitly imagine when they hear AGI, which is that humans are these sort of fountains of new possibilities of things to move to. When is the model going to be that? And that's a slightly different definition. Maybe it is the same definition, I guess, just applied more iteratively or recursively, I guess, of saying that when will it do truly everything and that there's no task at all. That we can imagine that humans will do. I think we're a ways away from being able to think coherently about that, of what does it mean to have no value in humans doing, say, philosophy, because all philosophy has been figured out better by machines. I think at the point that we start have to worry about AGI, like economic terms go out the window in some sense.

Okay. Just to take the other side for a second. Is there anything that you would say, like, how do you know if you're being too ambitious? If I want to say, okay, I've got this. task in my business and I've got a person doing it full-time. They're doing fine, but the volume is growing. They're going to have to hire 2 more people to do this thing, or maybe I can bring AI to bear on the problem. Is there any way, any framework or whatever to get to good judgment on, is this going to work for a given task? Like I can throw a million examples at you and you could evaluate them one by one. But is there any way we can abstract from that and say, okay, this is where you're reaching too far? Any common mistakes or any overreaches that you would warn people against?

I think a good rule of thumb is, you know, LLMs can do tasks, not jobs. If you have something that, that you can get down to a repeatable unit of a task, something where you can imagine the inputs and outputs laid out as text on a spreadsheet, that's potentially something that you can automate. I see a lot of novice users of LLMs will ask them for like a business strategy or something, right? What's an idea for a startup or whatever? I can't imagine any LLM is going to give you a good idea for a startup, anything like that, especially where it's current. You're not going to get that kind of originality. So I'd say that's probably one of the more common pitfalls is people asking for information that's in this sort of nexus of being recent and competitive. Like information where writing a funny joke is maybe one of the purer forms of forms of this, right? The LLMs aren't going to be able to tell you a good joke, except in the sense of maybe regurgitating one, right? Or applying something to a mold. Because humor is quickly, it changes so quickly, right? The things that were considered like world-class humor in the '30s probably wouldn't get very many chuckles today. So it's that like competitive landscape of people trying to usurp expectations and trying to surprise people. That just isn't what you typically get out of an LLM. Like humor is probably like the archetype of something that LLMs aren't going to be great at. And the less humour-like it is, the more chance you have succeeding, right? Something where you're not looking for it to be surprising and creative in a way that nobody's done before, where you're implicitly in competition with everyone else that's trying to do this task.

That's good. I often say precious few eureka moments from AI systems. And on the flip side of that, the more routine it is, the more we know what good looks like, the better chance we have of getting there. But that sort of narrows the scope to things where there is like a best practice or a standard of care or some sort of reasonable consensus as to what is good. If you don't have that consensus as to what is good, you're in trouble. Yeah, go ahead.

Though, to balance that out, I will add that I do sometimes see sparks of these.

Sparks of AGI?

Maybe not sparks of AGI, but like sparks of the connections that I feel confident are novel. Right. I had one, an example a few months ago, right in over here. It was on a YouTube video. I heard somebody like remark very quickly that the name of the Python library attrs, A-T-T-U-R-S, it was, quote, not a very nice thing to say to somebody in Dutch. And I wanted to know, what did that mean? Right? What were they referring to? And when I posed that question to ChatGPT, it got it right. It explained that there's like a Dutch insult called called etter. I think it was, or that would be like E-T-T-E-R. I forget exactly what it means, but it like pest or something like that. But it made that connection. And you can just logically, it's not going to see that anywhere because it was in a YouTube video unless they happened to transcribe that one video. And then like nobody was talking about this video, right? Like it's not something that it could have found out there. So I think like the, you do see these little reminders that like it's not entirely missing like novelty.

Yeah. An interesting challenge there is identifying the good ones. I remember the Google Paper Fund search where they set new state of the art on a couple interesting, kind of pretty esoteric from my perspective, open math problems. One was like a packing problem, how to pack these balls or whatever into a space in the most efficient possible way, that kind of thing. And this has been something people have worked on for a long time. They were able to advance the state of the art by doing huge numbers of generations. The key that they had in terms of an advantage is an objective function to score the little programmes that the language model was writing. And then they basically just iterated through keeping a couple of the best, highest scoring things and saying, hey, this is our best, give me something else. And they also did have one interesting strategy for trying to make sure they didn't get too trapped into a local maxima and try to make sure they would explore other parts of the the space, but they were able to advance the state of the art. It seems like a key thing there that they have, though, that we don't in general have, is an objective score on the problem. If you were trying to do humour and you were like, okay, I see sparks occasionally of good humour from the language model, I don't want to read a million bad jokes to find one diamond in the rough. I think the same problem, by the way, applies for tonnes of things, right? Like, people want to generate social media posts. They'd probably be happy to spend 10 times more on tokens to get one thing, or even 100 times more on tokens to get one good thing, if they could reliably actually select out the one good thing from the 100x more generations. So is there any best practice or any tips that you could advise there for when, like, okay, I'm willing to pay up to generate tons of stuff, but now how do I narrow that stuff back again to what's actually good? Do you trust the models to do a decent job of that?

To a point. So I think for, maybe not for humor, but for a surprising number of tasks, simply generating more and then taking the best of that through even like imperfect or crude methods can help a lot. So a very basic way of approaching this is to just generate multiple answers, pass all of them back to a long context LLM and say, which of these is the best? That's Maybe a little too crude, but there's more sophisticated things you can do. But what I would recommend most of the time is do comparisons rather than scores between different generations that you have. And from the comparison, the wins, losses, and ties, you can produce an Elo score and use that to determine which one is the best. This can really help a lot. It's hard to overstate. Like, so much of whether an LLM does a good job or not is happenstance and you want to just give it more attempts. I think there was a paper that showed that you would get roughly equivalent performance, I think, from doing 40 API calls to GPT-3.5 as you would for one API call of GPT-4. That's the exchange rate, right? So intelligence has a steep cost, but it does keep going to some extent, right? You can just keep generating more answers with GPT-4, rank them, and take the best ones. And then from there, there's refinements to that process that can help a lot too, right? Coming up with better procedures for ranking things. Trying to think how much I can share there. But even crude things do work. Even things as simple as just like asking the LLM which is best can help a lot. But wherever there are indicators of quality in your domain, you should use them. So if you're using code, like if you're generating code, you could use unit tests or you could execute the code and see what the results are. If you're doing translation and you want to know, is this a good translation? Backtranslate. Like, so go from the, from English to French, go from French back to English, and then have a model assess, is the English roughly the same as your original English, or is there some substantial difference that implies that a mistranslation happened? That's a contrived example because you wouldn't be doing this for translation, but apply to your own task. I think so much of the work of making LLM-powered systems better really comes from setting up these pipelines of ad hoc quality filters and checks and evaluations on your examples and making sure that there is no bad data in your examples, that you're not putting bad demonstrations, and that you're pulling in K-shot examples that really match the structure of your task. And the closer you can get to that structure, the better your performance will be. And often that change, that structure is trivial. So a more concrete concrete example of this, maybe if you're doing RAG, if you're trying to answer questions about a document, you are including context in your prompt that isn't in the form of a demonstration of your task. It's just added context from the document. It's snippets that were embedded similarly to your prompt. And there's all kinds of extensions like HyDE and things like that that go on and may have more elaborate procedures for retrieving the right chunks and different chunking strategies. But you're still making the LLM do a lot of work at inference time, right? That it's not just like picking the closest examples and interpolating between them. It's applying presumably novel question to like all these pieces of data. And that makes sense in a RAG context, but if you were like fine tuning, you would need to tune on actual demonstrations, right? So you would need to have tuning data that looked like Here's your input, which is some RAG chunks. Here's your question, and your input being the question and the RAG chunks. And then your output is whatever you get from those RAG chunks, right? And that demonstration can be generated synthetically. So like recasting documents into synthetic questions is a powerful strategy. And because also I think the underappreciated benefit of doing like a layer of synthetic transformation is that it gives you an opportunity to do quality checks, right? To start meddling with the sort of cached parts of the process, if you will, rather than trying to set everything up perfectly so that it goes right in production.

Yeah, those transformations, backfills, that's a really good tip. It's funny, I was doing early versions of that stuff in fine-tuning the never actually broadly released text-davinci-002 fine-tuning in the summer of 2022. 22, and you might have been doing the same thing at that time. But man, it was, in some ways it was like way harder. The models were way less capable. It took a lot more work to get anything to work. But in some sense, I think that era really prepared me well for now because I feel like that's actually more when I learned a lot of this stuff that we're talking about than today. It's like today the models are so good at so many things. Like, you can get tricked into, maybe I could just Tinker with the prompt on my way to get there or whatever. And when I think back to my 2022 experience, I'm like, this thing is going to do one thing, but I'm going to have to grind to get it there. And if I'm missing data and I need 1,000 examples or whatever, how am I going to create those? I don't really have time to create them manually. That's where a lot of these synthetic transformations and backfills and whatever became kind of part of my practice. And it's interesting that they're basically still the same core techniques are really operative now. It's just that We maybe have a harder time seeing through to the need for them in some cases because the models just off the shelf are so much better at so many different things, but they're still not good enough to actually be production ready for many of the things that people actually want to automate in their businesses.

Yeah, I think that's probably a good segue also to what you were talking about prompt engineering and how the profession has changed, or rather how the profession that showed up is not quite what was promised. I think in terms of scale, at least.

Yeah. So are you still a staff prompt Are you still a prompt engineer, or are you now an AI engineer? How do you think of yourself and what should we be expecting?

My title is still staff prompt engineer. It's anyone's guess whether that will be a title that people have a couple of years or if they'll rebrand it as AI engineer or things like this. I think there's a lot of ways things can go, but I think the thing that's clearest is that this image that was hyped up a lot in early 2023, particularly the sort of The job of the future narrative hasn't come to pass, right? That most firms don't hire their own prompt engineers, and there isn't a need for most companies, especially outside of tech, to be hiring full-time prompt engineers on staff. And I'd say honestly, like most of the demand for prompt engineering, it's coming from within the AI industry itself. That right now, one of the people who most need prompt engineering skills are people who are preparing training data, or not just preparing synthetic data from a model itself, but just filtering and doing quality control. And the applications of LLMs at a company like Scale or at any of the AI labs themselves are much more abundant than you see in many other contexts. I think that's the piece that has surprised a lot of people. And I never tried to lean into the promise that This was going to be a job of the future. That was never something that I really believed in, the sense that I thought everyone was going to do it. One thing I used to say a lot, and I think I maybe made this point on our last podcast, was I saw prompt engineering as a skill, something like being a typist. That it's a— it used to be that people who knew how to type could get a job knowing how to type, and that clearly isn't the case now. But everyone still knows how to type, right? It's a skill that everyone has. And people just don't call themselves typists, but they're still typing. I'm not even so sure if that's the right analogy now. I think most of the tricks and things that characterized prompt engineering of, say, 2022 are less relevant now. And they're certainly less mandatory. I think in 2022, there were many things that you had to know, right down to just really stupid things like not ending your prompt in a space. Before you could really get started to get good results out of like GPT-3. And that's no longer true. The chat models are just much more user-friendly. Many people get by just fine being completely unaware of tokenization, right? And they run into gotchas and they get bad answers sometimes because of it, but it's possible to not know. And that's probably the biggest explanation I'd say for what's changed about prompt engineering. And as we were like talking about earlier, that's why my Twitter output of like prompt engineering Tips and tricks posts to have declined lately. Because I think at the same time though, I guess you could call it like folk prompt engineering tricks of the things that fit in a tweet. Maybe the best known ones being like, let's think step by step and things like that. While those are less relevant and the ones that aren't obsolete are at least not mandatory now, prompt engineering hasn't stopped flourishing. It's moved into more complex areas. And it's becoming more and more code heavy. It's becoming more like programming than poetry. And it's also, I think, expanding into a lot of areas where we're not really calling it prompt engineering as much. I think people would debate maybe whether techniques like RAG are, if you squint a bit, RAG is arguably a form of prompt engineering, right? You're choosing what goes into the prompt. And you have, of course, optimization tools like DSPy. There's a lot of code and process being built up around like wrappers around LLMs. In a way that I honestly thought we would see sooner, but we're definitely seeing it now. This proliferation of tools, I don't see any sign of that slowing down. I think that prompt engineering has in some ways moved into less publicly visible task domains. We're now trying to solve problems in a way that the solutions can be bottled back into the LLM. Even back in 2020, I sometimes got the reply on Twitter of prompt engineering surely won't be a real thing because whatever there is to know, you can teach the model to do that. And that's not literally true for a lot of reasons. Maybe one day it will be like you'll actually have the model just using another model, but that's not literally how anything works today. But metaphorically, it is how it works today. That to the extent that we have tricks for Getting better output from LLMs, one of the best things to use those tricks for is to produce better post-training data and thereby eliminating the need for anybody to know that particular trick. And I think this is a good thing. We're going to see prompt engineering continue on this frontier. I metaphorically visualize it as something like the way a forest fire would move, right? The prompt engineering is the fire on the perimeter, but Any particular tree doesn't stay burning for very long. So I think like the frontier will continue. It'll just keep moving.

Yeah, that's funny. My version of that is the black hole that the models just suck everything into and eventually nothing can escape. That sort of sketches out implicitly like a vision of the future that maybe you won't even comment on this, but I assume that a lot of what is going on right now is creating the kind of multi-step, multi-app advanced reasoning traces for more agentic workflows, and that those ultimately are likely to get folded back into the models. And if I had to make a prediction of what is GPT-4.5 going to look like, if we are going to see a GPT-4.5, it would be yet another advance in the post-training where a lot of these stumbling points that we see with agent workflows today get addressed. Like, they can try a new way. They have figured out that, okay, you got stuck here, don't do the same thing over and over again, as we sometimes see today. And I think that could probably work pretty well even without that next order of magnitude scale-up in pre-training. I often feel like the models today are smart enough. I'm not asking you to tell me all the data that you're creating at scale, but I guess Is that like a, is there anything that you would say I'm way off on that expectation for what we might see in the next half step of models?

I think that sounds about right. I think there's entire avenues of things that we could train models on that we haven't really thought about much yet. And that's really exciting to me because it means that once somebody has the idea of lifting up some half-forgotten format of text and training an LLM on it, You can move into a new space very quickly, right? Like if you figure out DNA, right? I think we're going to see a lot of progress from that kind of exploration of just trying out new things to train it on. Or maybe another great example of this is that there was a DeepMind paper not that long ago where they distilled Stockfish into, uh, where Stockfish is being a traditional chess engine. And they created a chess AI. by this procedure, which is really clever, where they took essentially Stockfish's judgments of how winnable a board position is. So all the training examples were just like all of the form of like, here's a position of the board, how likely is it, do you think, that white will win? More or less. I'm glossing over some details. But the tuning on that is enough that given an actual position of the board, you can just say, what are all the legal moves you can make, run all of those through the model, say which one's best, and then do that. And that produces a pretty good chess engine. And so it shows that really that you can distill Stockfish's opinions of what chess positions are good into an LLM that was really built to model language. I think we're going to see a lot of those kinds of discoveries of that the knowledge there is in the corpus of all of these examples. Of all this training data. And collectively those imply an understanding of how to evaluate chess positions. So I think like we're going to see just more of that. That is like we just do pre-training, like discovering a lot of this stuff through happenstance. I think that's what characterized a lot of the 2022 era of LLMs is that LLMs hoovered up everything. And then we had this period of exploration where it was like, wow, look at all the stuff that's in here.

Yeah.

It can pretend to be Wikipedia. It can pretend to be the news. It can, you can write JSON and so on. These things weren't apparent at the beginning. And I think that's what we're going to find new things to put in.

Yeah, I think for me, it's like you can ask this question of, are things going to level off? Are they going to level off? From my perspective, it's to not get to some definition of AGI where maybe we still don't have too many eureka moments coming from. AI, but where they can do a huge percentage of just the tasks that people get paid to do at work most of the time, most of which are not eureka moments, but are doing a task and trying to do it reliably and consistently. It seems to me like for us not to get there, things would have to level off really quick. And that seems almost impossible, even if we just imagine training on a lot more expert data or pristine data, as I've heard your CEO refer to it. But then on top of that, maybe it does kind of level off at expert level. And then I'm like, nah, there are all these other modalities. That's where I just can't see how it doesn't all come together, because the Stockfish example is a great one. DNA that I've talked about many different episodes with models like Evo is another great one. I just did another episode on predicting very small-scale molecular dynamic evolutions where this dude, Tim Dagnan has had this super viral tweet where he's just simulating small salt solutions, literally just like salt water, but observed that he was spontaneously seeing crystallisation happening in the simulation. And he was like, we didn't even train this thing on crystals, but we just trained it to solve the wave equation at any given moment. It's very similar to the Stockfish thing where it's like he has a computationally super intensive approach to solve the wave equation. And he's just training the model to make its best guess as to what the real solver is gonna do. And that turns out to be accurate enough that he can run essentially the same simulations at like orders of magnitude speedup and start to see these things that would never have been possible to see before. And it's crazy. The scales that they're talking about are like small number of atoms. It's like picoseconds level. Resolution, but seeing these things like, oh my God, crystals are forming in my simulation now. So when I see that, I'm like, man, there's a lot more modalities out there besides, there's a million different kinds of sensors in the world. And all of these things seem like they're destined to find their way into the same ginormous pool of data. And with all those modalities, and then you put in like a Sora, good God, you've got intuitive physics, then object permanence popping up. It seems like We are definitely headed for something that is, at least in some very meaningful ways, superhuman. That doesn't necessarily mean it'd be superintelligent, or it doesn't mean it would have no weaknesses, but it does seem like there's a pretty clear path to me at this point to something that is in many ways and in many meaningful ways a superhuman system.

Yeah, I agree. I think people have the impression of the data-centric view of AI, of The hidden in the models is data, as they say. View is sometimes, I think, seen as like the skeptical or conservative position on like capabilities, right? That it's the one that, that jives with the worldview of these models are rising only to the level of intelligence that we train them on and no further, right? That's like a skeptical position that you sometimes hear people advocate. And so you're not going to see any progress beyond what we have now because this is just like as far, this is as good as what humans can do. And that's All we get is what we train from, what we get from training on that. I don't give much credence to that position I have of seeing models as compressed data. In some ways, it's really optimistic about what they can achieve because I see this sort of fungibility between thought and experience. And I think we're seeing that theme in a lot of ways, like all the way back to just the idea of generative pre-training, that the task of token prediction and Like, just feeling out the vibes of what token should come next in a statistical sense leads to something that resembles reasoning, right? That it resembles step-by-step thinking and calculation and all these other things that we could at least mistake for thought. I think that pattern is going to continue, right? That we'll see things, more things like the Stockfish distillation example. We'll find that there's just more and more kinds of knowledge and experience that can be put into weights. And I think that's really exciting, like you said, that even if scaling were to somehow magically stop, if a government comes in and says you can only go up to this many parameters or whatever, there'd still be a lot of progress, I think, just from training on better data. Even on the pre-trained side, just knowing what to pre-train on is still very much a dark art.

Yeah, that GEST paper had a Google in just the last week or so, I thought, was another, oh my God, this is a big advance. And even if it was the kind of, that was the sort of thing where even if you discount how much you want to believe in it, you could discount it quite a bit and it could still be like a pretty big deal. I don't know if you've studied that one, but for listeners, if not for you, the idea there was that they took a small model and trained it on the same data that they're considering training the large model on. And then look for data that is not yet learned by the large model, but is learnable in the sense that the small model at least somewhat makes progress on it. So it's essentially separating what the models can learn from what they're not learning much from, and then selectively training the large model on the more learnable data. And they report just your casual 10x efficiency improvement from just that one neat trick, which is crazy too, because that compounds with many of the other things that you might think are going to be great, but you could also apply that to, and it doesn't seem like it's an indirect competition with other improvements. It's important, I think, to be clear, the compute thresholds today are for a reporting requirement only, right? They're not actually telling you, like, from the executive order, they're not telling you you can't do it. They're just saying you have to let the government know if you are doing it. So I've felt like that's a pretty good idea. And I've generally agreed with the idea that, first of all, if somebody's doing multi-hundred million dollar training runs, yeah, probably doesn't hurt for the government to at least have an awareness of who's out there doing that. And then also, I can't come up with a better threshold, but the more you see all these just efficiency gains coming from everywhere, it's, man, even that is, it's a pretty hard position to really think, oh yeah, we can just set this threshold. And then as long as everything, we know everything above that threshold will be fine. It's happening faster than I would've anticipated that you take 10x out of the difference between 10 to the 25 and 10 to the 26. It turns out to be quite a bit. It's 90%, which is, it's funny how that works. But anyway, there's a lot happening. I think we both expect things to get pretty crazy. You mentioned DSPy briefly. Is that something you use a lot? And in general, is there like a best overarching automation system for the sort of work that we've been talking about through most of this?

DSPy, I've used it. I wouldn't say that I've used it a lot. And that's not really a reflection on DSPy as much as like limited time that I've had it on my, my backlog to get deeper into it, to find more use cases for it. But I am excited about the project. I think like there's a lot of good ideas in there. And I think there are a lot of niches where automating prompt engineering makes a lot of sense.

On the dark art of prompt engineering. the jailbreaking side. I'm sure you're a— in some ways, maybe your Twitter spiritual successor might be Pliny, who is jailbreaking every model in 2 seconds from the second that he can get access to it. It seems like he's usually using the same techniques across a lot of different models, and they basically just work. My question for you is, do you think that those techniques will eventually be stamped out of the models through more reinforcement training or whatever? Or are we headed for a scenario where the model itself just can't get that robust and we instead have to have some sort of moderation layer or governing system on top of the model to control these sort of jailbreak type strategies?

There's a few questions in there. So first, regarding whether you can do it entirely in the model through training, like better data versus filters. There seems to be increasing evidence that there are cases where you can't do everything entirely in the weights. So Anthropic had a paper not long ago on mini-shot jailbreak attack, where you basically repeat hundreds of examples of malicious prompts and responses into a long context model. And this overpowers fine-tuning-based defenses against this, of tuning on adversarial examples of it behaving correctly. That same paper, though, found that a pretty reliable defense against this, which was to simply add a suffix to the prompt reminding the model that it's a safe model, that it's Claude, and that it's well-aligned and all that. And this was remarkably effective. I forget exactly what the numbers were, but it was very effective at stopping the behavior that they were showing. And that's a very simple trick. So I'm sure you're seeing things like that one today. But beyond that, I think we shouldn't be— we shouldn't trust blindly that there's going to be like a tuning fix for all these things. I think there's, there's, there's very likely going to be a place for things like content filters. There already is, right? There are good examples of this, I know, in like the token repetition attacks. So it's trivially simple to have a filter on your API that just says, is the user trying to repeat the same token over and over again? And if you have an LLM continue a prompt that has too many repetitions of the same token, you can get into this mode where it will probabilistically spit out pre-trained data. There is a paper that showed this by collecting lots of this data and indexing it all together. Well, so for that particular example, I wouldn't say strongly that there's no way to do it in tuning, but it's hard and it's not worth it, right? You're much better off just like handling this with filters on the input to protect against that particular issue and filters on the output to look for tricks that try to get the model to do it itself or filters for the copyrighted content that you're trying to protect. But it's all a cat and mouse game. I just, I think that. We spent a long time in the regime of trying to do everything through tuning. And I think we're finding that there's like holes in that strategy, that there are some like just odd cases where it doesn't make sense to, to patch everything like the same way. And so I think like we're going to see, yeah, more of that.

There's a follow-up though, which is like, what, if anything, does that imply for the future of open source? It would seem to suggest that we are either headed for a world where people are going to say, hey, these models are too powerful to be distributed in the raw. They have to be somehow contained in an envelope that keeps them from doing unwanted things. Or we may just say the value of open sourcing them, which is certainly very significant, is just so high that we'll have to live with the downside risks of that. Do you see any other way to not have to bite the bullet on one side or the other of that debate? It would be really nice if there was some way to be like, yeah, we can have the best of both worlds. But right now it seems like we're headed for having to make a choice.

Yeah. Once you release the weights, people can do what they want with it. There isn't really any method that I'm aware of that you could have a, an open source model without like damaging, like attempts to censor pre-trained data to make a, if you want to really squash the problem, you can make it unaware of concepts. Right. But that has downsides as well. So that's becoming, I think, less popular, starting with Llama 2, of filtering out just concepts entirely from the pre-trained data. So I think there's no free lunch there, but as to what are the solutions to that problem, I'm not really sure.

Yeah. One that this is just making me think of that you may have seen is called SOFON, and it's out of a Chinese lab, actually. It's a strategy for fine-tuning suppression. And this is not on large-scale language models. It's on smaller scale, like image generation models is where they started. So there's definitely some work that would have to be done to generalise this. But as you of course are well aware, the image models typically work on a denoising principle. And what they tried to do is create an image generation model that would denoise as they normally do, except in what they call the restricted domain, where in the restricted domain it would just output basically the null transformation. And so it would essentially not denoise. And they crafted the loss function in such a way where it's like smooth-bottomed. So they fine-tuned it with this combination of, if you're outside of the restricted domain, just do your normal thing. If you're in the restricted domain, you're supposed to produce a zero. And the loss bottom has a smooth curve such that it's hard to fine-tune out of that local minima, because the gradient, once you converge on this kind of smooth bottom loss function, the gradient is zero. And so if you're trying to fine tune out of this local minima, it's hard to know where to go. And they found that training these models in this way created a model that is harder to fine tune to do the unwanted thing, as opposed to just training from scratch to do the unwanted thing. So I thought that was pretty interesting. There's very few of these proposals flying around that I've seen. I'm also pretty interested in the Dan Hendrycks et al. line of work around short circuiting, where they take representations and essentially do internal classification. And they've actually just recently started their company on this premise as well. Those are a couple of things that are on my radar, but As of now, we don't have answers. The short circuiting, that's still more of a system kind of approach. The Sophon thing is interesting in that it's fully baked into the model. And so you could imagine if that were to be really generalised and scaled up, you could imagine something where you might make a sufficiently fine-tuning resistant model and not have to bite the bullet on these trade-offs. But we have a long way to go on that. Anyway, I'd like to highlight that work because that flew way under the radar. Zvi mentioned it to me in a podcast, and I hadn't even heard about it at the time. And it's out of a Chinese lab, so it's definitely one that I think deserves a little more sunshine.

Yeah, it's the first time I'm hearing of it too.

So fun. Yeah, it's good. Or at least it's super interesting anyway. We'll see how far it can be pushed. Okay, state-space models. Are you a Mamba stan? And what do you think the promise of subquadratic architectures is and will mean for the field as a whole?

I'm really excited about it. I watch these sorts of things as an interested outsider. I'm not really qualified to say much about architectural decisions for experimental architectures for new LLMs, to be honest. But I have some familiarity with state-space models for time series modeling. Before I got into LLMs, I was an ML engineer working on massively multivariate time series prediction. So I've, they're, they're near and dear to my heart in that sense, but it's not something I've like done much work with recently. So I'm excited, but I don't really have too much insightful to add there.

Cool. Last one. What do you think the prospects are for solving the ARC challenge if we gave you a version that is no budget constraint and just said you just have to make it work and you can fine-tune any of the models that are soon to come online for fine-tuning, like GPT-4o is in fine-tuning beta now, and Claude and Flash are announced, all of the tools that we've talked about throughout this conversation. Do you think you can get there with Arc and create something that can actually hit that sort of human-level performance? Or do you think we're still just not there in some fundamental sense?

Yeah, Arc's really interesting to me. It's satisfying to me that it recapitulates the pattern we see in human intelligence testing, where you see things like that. Raven's Progressive Matrices, which is like an IQ test that most people wouldn't recognize as like what they think of as an IQ test. And that it's a pro— it's a test that has no words. Every problem can be described as, here's something, uh, here's a pattern that resembles like wallpaper and there's a piece missing. What piece do you think is missing? And then you have like multiple choices.

Mm-hmm.

But you can have all sorts of like logic problems that are expressed in this form. And so it's interesting that we're having something like that for LLMs that you have something that's like the other approach that you could take to intelligence testing, like the SAT approach almost, of how much vocabulary do you know? How many things do you know about the world? How much do you know about writing? How much do you know about music? And then you could test everything under the sun. But maybe this is not the right way to view the world. It's a bit of envy of the simplicity of physics maybe. But it feels right somehow that there could be just a simple task that captures this deeper notion of generalizable intelligence that we think is there. But I'm unsure whether it's real. We spent a long time trying to think what are the grand theories of intelligence, just armchair philosophy on like how do humans think and trying to tease it apart and capture that in symbolic AI and all the other attempts at AI over the years. And the thing that worked is just making it know everything, right? There may be something irreducible. about that, that maybe there's a body of knowledge that has to be there for the thing that we think of as intelligence to even be coherent as an idea. But its possibilities are wide open. And I think the fact that we can be sitting around just speculating this widely about the future really says how little we know about this stuff.

Yeah, that's maybe a great note to end on. And I think a very good reminder. Is there anything else that we didn't talk about that you want to talk about? And this is if you want to share more about Scale in general, or eval stuff you're working on there, or the SEAL thing, or product-wise, basically floor's wide open for anything that you want to put some extra attention on.

Okay, great. I'll just say that Scale is hiring prompt engineers. If you're interested in prompt engineering, if really any of the issues we were talking about here today, especially if you're interested in LLM security and red teaming and jailbreaking adversarial prompting. We're looking for people who can do impressive things in prompting. Reach out to me on Twitter or through our SCaLE job boards. But I'll do my best to make sure, you know, you get sent to the right places.

Anything people should do to demonstrate this? Anybody could say, oh yeah, I love using ChatGPT. How can somebody distinguish themselves? What kinds of things do you look for? What stands out to you that's impressive so that people are not just yet another resume that's like enthusiastic about ChatGPT?

Yeah, a lot of people are talking about Pliny. So things that go viral, it's an art unto itself of like just seeing what's getting attention, what people are liking, but that's, it's a good way to get to the top of the stack. I think if you can let X, the everything app, sift through the quality ideas and let the good things bubble up. It's the avenue that worked for me, so I'm a little biased. So take it with a grain of salt, I guess.

This has been fantastic. A lot of really good content here that I think is just going to be of super practical value to people as they seek to use AI to actually automate work and get real economic value. I know you have been hard at work figuring out which among these techniques is really worth it. And also happy to say that I think our views of this definitely are converging on some sort of platonic approach today. So that's been a great finding for me as well. I really appreciate the time. I think folks will get a lot of value from it. For now, I'll just say Riley Goodside, the world's first staff prompt engineer at Scale AI. Thank you again for being part of the Cognitive Revolution.

Thank you so much. It's been great.

It is both energizing and enlightening to hear why people listen and learn what they value about the show.

So please don't hesitate to reach out via email at tcr@turpentine.co Or you can DM me on the social media platform of your choice.