Fireside Chat with DSPy Creator w/ Omar Khattab

Anyways, let's get started. We are very lucky here to have Omar to come give a fireside chat. So I have a fire and I'm beside it. So hopefully it's not all in vain. But Omar is— give some background. Omar is a— I guess now he started his assistant professorship at MIT in the EECS department. And he has done a lot of work in the open source community. Around evaluation, building what's called compound AI systems. He has a great blog post about that. And I thought what was also very interesting is before that, he worked a lot on retrieval. If folks are familiar with the ColBERT model, Omar is behind that. So Omar has a lot to do with pretty much state-of-the-art technology that we hear and use today. So with that, very excited to have him. Hamel, anything else you wanna add to the bio and background before we get started?

No, I think you covered it really well. I am also a near fireside Portland style. And yeah, I'm really interested in this talk. DSPi comes up in every single conversation. Not just about evals, about AI engineering. I can be walking down the street and just say the word, say anything about LLMs, and it feels like there's a guy that pops out behind the bush just like at that moment and says, what about DSPy?

Every time, like, this is the level of fanfare of DSPy is unmatched.

So Omar, We just have a set of questions that we have for you, but you can use this as a soapbox to go on whatever rant you want. But maybe you want to start by telling us how you got into what's modernly known as AI engineering, a little bit of background on yourself, your— whatever you want to talk about, really.

Yeah, I mean, first of all, thanks guys for hosting me, and I've been a fan of the work you've separately been doing for a long time, but also the course and the book. Hamel shared a copy with me a while back. It's great. I love it. And honored to be here and talk to all these amazing folks in the community you've gathered together. So I'm not really sure what counts as AI engineering once you, like, the farther you go, the harder it is to define, right? I think it's becoming actually a pretty concrete thing now. And it's, it's, it is clearly a separate, like, important field. Without making this too long, let me give you context that will help actually the rest of the conversation. So I started programming like many of the folks here around age 11 or 12 or so. And I had two things that I was so fascinated with that I didn't understand at all, but were bugging me. And they're the two things that are still bugging me to this day. The first was search, because if you think of AI, like Google was maybe the first, I don't know, for many of us, like the first thing that looked somewhat intelligent and was a black box and you had no idea what it was doing, even though it was like mostly just a bunch of features and text matching at scale. And the other one was like, I loved programming. I loved loops. I loved functions. I loved conditionals. I had no problem with those, but I was confused as to like why I can't be slightly less precise when I want to be slightly less precise. If I have a piece of text and I say, is this spam or not? I just had no clue how on earth to express that. Like nothing in the primitives I'd learned, like, okay, I could do nested loops, I could do regular expressions, I could do SQL. Like nothing seemed like it could, it could figure out like how to give me a function that could do that. And I had never heard of machine learning at that time, or like after that, like deep learning, all that, like maybe even the keyword deep learning wasn't really particularly mainstream at the time, right? So I just had no idea how to access that. And I just spent a lot of time thinking, trying to think of like how my programming language could just understand my intent a little bit better. And I think, so this is my angle on AI, and I think it differs from a lot of people's angle on AI, which is like, they're thinking of, you know, agents, they're thinking of robots that are smart, they're thinking of like being able to just ask for things in high-level way. And I, to me, the angle was, no, I love programming actually. I do want to write code, but just sometimes the code I want to write is not very well specified. It's not like perfectly precise or specified. So then I started my undergrad and I started doing research on AI systems broadly. So this was a lot of distributed graph systems. That was the kind of popular thing at the time. And the focus was really on making them really fast without sacrificing too much quality. Like a lot of people in the systems community were spending time doing that kind of thing. And what fascinated me there was like how clean the programming models were, like such that a whole lot of algorithms slot right into the programming model and you could just build like really fancy systems that hide a lot of complexity. But I felt really uncomfortable because the programming models coerced you to think in a certain way and it wasn't clear, are we building algorithms that the programming model likes or are we building programming models that allow us to express what we want to express? So I was very uncomfortable and I started working on search, on information retrieval. And that was like the loveliest switch of my life because that's like the thing that always fascinated me. And this was before BERT. So like back then it was still text matching, it was still variants of BM25. And when you don't have BERT, like you're obsessed over all the different variants of like TF-IDF and BM25. Like there's a gazillion type of language modeling scoring functions and stuff of that nature. And what I loved about IR was, I think it's like the original— IR and recommendation, I think, are like the original AI engineering, in my opinion, because you're building systems. They have to be systems. You're not building— you're not like just building general like classifiers or models. You really have to get something that like will sit in front of a user. The definition of success is a lot more than like, you know, some F1 score maybe or something. It's really sophisticated, right? It's open-ended to a degree. If you think of the output of a retrieval system, it's much more vast than a classifier, but it's a bit more constrained than a language model. And I spent a lot of time thinking about how to make these systems faster, and I built like a couple of nice frameworks in that space and whatnot. Then BERT came about, and I started my PhD at Stanford. And so it was really interesting because I spent my undergrad trying to make retrieval run in 3 milliseconds instead of 30 milliseconds. And then BERT was like, oh, actually, do you want to wait like 30 seconds for the query and get 10% better quality? And I was like, what happened to this community? Like, we've been trying to squeeze out like 1 millisecond a year in speed. And then people are like just orders of magnitude slower. So I worked on ColBERT as a way of like, how do we take something like BERT and then scale it to a large collection without giving up on any quality? It's basically the same quality as if you were scoring every document with BERT. And so that was like the late interaction approach. And what I loved about ColBERT was it was, I don't know, I imagine folks here are really familiar with the Bitter Lesson. So NLP at the time, like the field of natural language processing was going through, like, we've been having to contend with the fast pace of AI for the last, I don't know, 10 years, a little bit longer maybe, where like every year, every couple of years, the models we work with, the approaches we work with, like everything just keeps getting reinvented and changing quite dramatically. So like BERT had just made a whole lot of stuff seemingly redundant potentially. And then we were getting like all kinds of other new models, RoBERTa, GPT-2, T5. And so people were like wondering, how do we build— what are we supposed to be doing? Other than waiting for the next better foundation model. Maybe the name wasn't even coined at the time. So what I really liked about ColBERT was it is a model, yes. So there's a checkpoint and a lot of people use that checkpoint or the different checkpoints, but it was an AI system primarily where maybe the question was like, how do I encode billions of documents and how do I search them? And like the foundation model was really just this swappable piece. Such that you get a better swappable encoder and you get a better ColBERT. And so to this day, there are things like ColPALY, which is the ColBERT that is multimodal. There is, I don't know, Hosted Antoine, who talked about reasoning-based modern ColBERT and variations of that. So there's this essence of an interesting idea that is as orthogonal to the base model almost as conventional software is orthogonal to the hardware you run it on, right? If you write a piece of C code or a piece of Python code, like the amazing thing of Moore's Law for a long time was you just wait for the next generation of hardware. You don't even think about it. You just run your thing on the new computer and like it just runs. But not only does it run, it's a lot faster. It's a lot nicer. You have more memory. You can run bigger scale stuff. It's just like this amazing gift that like computer scientists and software engineers like took for granted for a very long time. And it changed or stopped like some 10, 20 years ago in its trends. And so like we had that contrast between the way we do programming where like we're trending towards, at least for a long time, we kept trending towards higher and higher level languages where you say what you want to say to a degree, except of course that you had to be really precise that you couldn't get rid of that limitation in some sense. And then the better hardware and the better operating systems and the better infrastructure just kind of served you and didn't get in the way too much. And then we have on the other end AI systems where, I don't know, every time it changes, you trash all of your work essentially if you maybe are not— if you don't mind doing that. And you're like, ah, it's a bit of a mess. We're struggling under the improvement rate of AI. An interesting contrast. So after Colbert, I started thinking about, all right, What can we take from IR to help the rest of the NLP ecosystem? So that was early 2020. And so the ideas in vogue at the time were language models will just keep getting better. We'll just scale pre-training. It's all unsupervised. They're just going to be able to do everything. They're going to memorize everything. And don't try to do NLP where you modularize stuff. Just wait for the better models. And so I spent a lot of time building things like ColBERT-QA, which was happening in parallel with the RAG paper. It was released like a month later or 3 weeks later. and it was a lot better in scores where we were basically doing retrieval augmented sort of question answering, but with a ColBERT piece that is being trained in interesting ways. And I built systems like Beilin after that, which was all about what is now what we now call deep research. Like how do you take a question that is complicated, break it down, search for a while, collect some sources, summarize them, search again, and these types of, these types of pipelines or agents. And so that kind of showed me the, But it is actually possible to think about architectures that are pretty independent of the language models themselves. And in a way, it is information in the world, information in the task that you want to capture that can complement the model you're working with. And it was specifically like the most interesting piece was this information flow of, hey, I'm going to give you a tool such that you can search, or actually I'm going to train you to generate embeddings such that you can search. You can access these specialized documents. They have information that you might not have seen before because maybe they're too recent or they're internal or something else. And you just give me a better model, I'll slot it right in. Models at that time were not very good. So in order to actually build architectures like that, you had to train every step to do its task. So in Berlin, there were like 6 models. You had to basically train a model to summarize, a model to act as an embedding generator for retrieval. You had to train a model that would generate, act as a retriever for future hops, like conditioning on previous hops, right? So a lot of the things we take for granted with prompts had to be specifically trained. And often you would not have data for every single variation of step you want. So you would have to do this sort of thing where you define evals end-to-end. You say, I want the system to successfully retrieve this document, but I don't know through which path. So I will basically reward it for doing the right thing. So I'll basically let the system run. And if it does the right thing, I'll basically use that as a training data point, essentially. So this is like the sort of self-training stuff that looks like what we now call reinforcement learning with verifiable rewards, except in much more primitive models and much more, with a lot more handholding. So then language models improved and we got things that were prompting oriented and they were like much larger, orders of magnitude larger. So we had GPT-3 was, I don't know, like over 150, 170 billion parameter models. The models I was working with before that were 1,000 times smaller. But it was very clear that the same architecture you built, the same sort of multi-step decomposition, the same access to the tools, the same access to the corpus, wasn't supposed to go anywhere. You just had to express things in a completely different way. And that really bothered me. So that led to DSPy in early 2022, where we were thinking, could we think of a, like a language essentially, not like a programming language, but some kind of embedded language or some kind of framework where you can just say the steps you want. And those steps are independent of the paradigm in machine learning at the time. If it's prompting-based, if it's fine-tuning-based, if it's reinforcement learning-based, if it's some ancient thing. If you go back to early NLP literature in the 2000s, like 2006, people were still building these modular systems that could do search and could do translation. And then they string them together. And you could get a Japanese question-answering system that uses English documents. You could always have these modular decompositions. It's just that the pieces were functioning very differently. So these were the 3 pieces that I learned over the years. They were information flow that you have when you're trying to describe an AI system is something that only you know. And like the context, you know, what people might refer to now as context engineering, is really actually something that like is pretty unique to your application. It is really important to be able to decouple defining that flow from whatever technology is in vogue at the time or is best at the time. And it's really important to have ability to align the model with your system after you've specified the whole system. And that this can take a lot of different forms. So this is my journey entering into the era that is, is AI engineering.

Yeah, thanks for the great overview. And I think it leads very nicely into Skylar, who has a question on the Discord. He pointed out that you have multiple successful research projects. So do you think that there's something unique about your research approach that leads to such bigger excitement? Yeah, how do you think broadly, do you feel like you have a research philosophy? In that sense.

So I value open source. I really like open source and it's something I deeply enjoy. I think also one thing that is a source of, source of frustration, but also a source of good ideas is I'm very opinionated and like I start with how I want things to look, not with what the technology looks like today. And so like, I'm very comfortable letting a lot of the mainstream, more mainstream stuff diverge from where I am, and I'm not going to rewrite what I'm doing in the new words necessarily. I'll relate to them, but I think this just colors what we do. And so like, I'm very comfortable, for example, and this is a bit of a tangent, but it's actually, I think it's going to be helpful for maybe like for today's audience. Like I'm very comfortable going off. I've been doing this for years where like DSPY is often understood by a lot of people as a prompt optimizer, and I'm very comfortable writing very long threads every other week of, no, we're not a prompt optimizer. Prompt optimizers, as people often understand them, are not a great idea. But what DSPY does is, I think, a great idea. And so I'm very comfortable with nuance of that sort. And maybe actually let me touch on that briefly, because I think it is—

Yeah, I was going to say, can you jump into, okay. How would you define prompt optimization? And Hamel Husain and I get from everybody, like, when do I do prompt optimization? Can I do it now? Most people have not even read their prompt or their outputs or know even what their inputs are, and they want to jump to prompt optimization. So can you talk first, maybe define it, how you think about it, and then when is it appropriate to do it?

So I think, so The motto for DSPy is programming, not prompting language models. And it maybe should have been programming, not just prompting language models. And so the idea there is in order to just declare what you even care about, in order for you to like just let the world know what you want, you need 3 things in the general case. Sometimes you can get away without some, without all of them, but in the harder, usual, like general cases, you need 3 things. Those are you need code, you need control flow, You need to be able to say, hey, here are my tools, here's my structure. I want this module to see these things, but not those other things, because it doesn't have to do with them. So you need that control flow. And for that, like, there's a lot of ways to do it. You could have it visually, you could have it in other forms. To me, I like programming, so I want this to just be like literally Python or some other language. And you need to combine that with natural language specifications. And I think a lot of people approach DSP sometimes successfully, thinking that they're not supposed to do this or that they're not supposed to engage in this. And that's not correct. Like the whole power of this new paradigm is that you will have fuzzy pieces that you are best equipped to declare in English and you should put them in English, right? So like you have a functionality that is supposed to summarize in a certain way with a certain tone. Like just say that. Yeah, that's really helpful. That's a great capability. It's core to this paradigm. And then there is the question of, Well, how do I get my language model or whatever technology I'm working with, but it is a language model right now in a certain way. How do I get it to do well? And this is the long tail of concerns of, oh, it does badly on this question. Oh, it fails under this edge case. It handles— this is like a long tail of stuff where it's not that you didn't ask for all the right things. It's just that like, it's not living up to the expectations that you have. And so this last mile, that last mile is a problem of optimization. And the point that DSPY tries to make is— and by the way, this is something that I found really beautiful in the course here, which is you guys defined the three gulfs. It's a beautiful, I think, analogy or structure that really is consistent with how the DSPY paradigm views this thing. So there is a question of you iterate in the outer loop to even just understand what it is that you want to specify, right? I think— I don't know what the word is, but it's like a form of discovery, right? It's like you're trying to understand—

specification.

Um, no, I think it's—

we have Gulf of Comprehension.

Yeah, yeah, yeah. So this is just understanding the problem, make sure you align with the stakeholders. Maybe it's just you and you just really want to define it, is what is it that you actually want. Then there's a question of like, all right, let's say I just actually put that in a form that like I can read in the future, that my editor can see, and that editor has like the IDE and that a future programmer or future me can see in the future. This is the ground truth. That is the intention here. And then there is what you call the gulf of generalization, which is the problem of optimization that we have where you're like, actually, I'm not adding any new information in the system. I may be repeating myself in a certain way. I'm using all caps. I heard about this really cool trick. There is role prompting. There is— I'm actually switching from JSON to XML because I heard that works better. And maybe it does, actually. All of these things. Like, they're funny, but like, they actually can help. It can actually be the case that switching to XML helps your application. Actually, that's the whole point, maybe. And to me, I think as a community, just from a, from a long-term standpoint and from a complexity standpoint and just from like organizing the collective efforts, I think I just have this argument that we should invest in this the way that like we invested in compilers in early computing. 'Cause if you think of someone programming a machine with, I don't know, 8 kilobyte RAM or something, and they're writing assembly and they're like, they're really working with that one machine. They're not gonna change it for 3, 4, 5 years and whatever it is, right? So they're really diving deep and they're avoiding like assigning things in that way. They're changing the order of the instructions in a certain way and like they really master it. But you get a better machine, maybe the assembly's slightly different, the tricks are slightly different, and then we're stuck now with this like, prematurely optimized, over-engineered system that is hard to work with. And so like, I think the beauty of higher-level languages, it's not precisely that just that they're faster, although as technology gets better and as compute scales, the better lesson is that they actually get, they actually perform even better. But the beauty of it is you say what you actually want in the language that you speak. And in the case of, in the case of AI systems, it's code, it's English, and it's, so code here is structure. It's English and it's evals. And so like you can't really in the general case do without any of these three. What evals gives you, automated evals specifically, is the long tail of problems that you play whack-a-mole with normally by editing prompts and seeing what happens. I think the actual intention you have is to do well on them. Your intention isn't that like putting this instruction at the end instead of at the beginning works better. That is going to be my prompt forever. Because if you change the model, if, I don't know, the specification starts to change, if you hand over the code to someone else and they have to maintain it, none of that is documented anywhere. It's just a trick that you applied and you're stuck with that particular manifestation there of your intent. Back to prompt optimization. I think that what a lot of people want and what we like are so comfortable telling them, that's not what we do. Like, That's, I think, one reason that I love that DSPy is just an open source academic, academically oriented framework, although we have a lot of users, is very easy to say no. Because I don't know, people are not going to pay us something or whatever. And I think that is actually making positive change. So what people often want to do, obviously not everybody, and I think a lot of people bypass that stage, is they're like, I want to give you— I have a bad string. It doesn't work. I want to give you my bad string. and I want you to give me a better string because I'll copy that better string and I'll go on with my life. And we're like, nope, we're not going to do that. If you want to use the optimizers, which are great, we love them, we're proud of them, you got to write your code in a certain way. And the point of writing your code in that way is you're going to write your signature where you're like not going to couple a bunch of things that you shouldn't couple, but you actually are going to couple a bunch of other things. So what you, what we coupled together is that we, you have to work in modules. And I've seen a lot of code where people would be like, here's the code that like combines the strings, and then here's the code that calls the language model, and then here's the code that does the parsing, and they're separate. And in DSPy, that's not allowed. Modules handle like that. It's a function. It takes objects and it gives you back objects, structured inputs, structured outputs. And these two are separated from the definition of the module. So we've coupled certain things and decoupled certain things, giving you a function. So now the language model is a processor of that function that is like a separate object. It's also defined in a separate place. And what that means is that you're not concerned with a lot of things. You're not concerned with how do I take these objects and actually materialize the right sort of structure of the prompt. That's not the job in the, in defining the function. You actually control that, but you control it in a different place called adapters if you want to change it, if you end up needing to think about it, which usually you don't. And in doing that, the signature also does not specify the output format. You don't say, gimme JSON. You don't say, gimme XML. What do you mean these things? Like your goal is to get back the objects. So a beautiful thing about this is if you have a product manager or if you have somebody who just wants to define the specification, that is included in the signature as pure instructions that simply define the task and not some other lower level concerns. Now when you do this, what you've done is you've separated the structure of the inputs and structure of the outputs from the task definition, from the actual calls to the system. And what that means is there's a lot of algorithms that get unlocked this way. So what that means is if someone says, hey, it's the world of reasoning right now, and we want to apply reasoning to our prompts. Now, this is not relevant anymore because models are now reasoning models for some reason. In DSPY, you would just say, all right, apply a chain-of-thought transformation to the signature where that is like the signature does not know what a chain-of-thought is. It's applied on top. If someone says, hey, I want an agent that has the following tools, it's not the job of the task to know that it's an agent or that it has tools. The job is just like to translate or to summarize, right? So you can apply that transformation on top. There's no optimization here. This is a genetic function. That you're defining. And lastly is the problem of optimization. So now you have a system and it runs, and you've iterated on the programming and you've iterated on the evaluation, and you've built an evaluation set. You have 100 data points that you're comfortable with, you're measuring the right things, you've done the error analysis, you've defined the system to the best specification that you have, and then you swap models, and then all of a sudden the system is not working very well. What DSPY allows you to say is, here is sort of a menu of optimizers you can choose from. They can build few-shot examples. They can iterate on their instructions. They can do reinforcement learning with GRPO. They can do reinforcement learning with filtered behavior cloning. Try them. Simply run them overnight or run them while you do something else. There's a pretty good chance your system is going to be a whole lot better, or you're going to know if there is not a whole lot of room. To, in the kind of final mile of alignment. But in all of this, your code is saying what you want it to say, not what like a particular model decided that it really wants to hear.

So I have two questions that come up for me when you're talking about this. I agree with everything you said, by the way. Love the shout out to the Gulfs, obviously. Um, but first one is you— there's a problem that I face where like AI models are changing so fast that sometimes I wonder Do I need to fully change my specification even though my task is the same because the models have changed? And you made a great example of I could add one line DSPY chain of thought, but if models get so much better, I might actually want to remove that from my DSPY pipeline or specification. So can you talk a little bit about like how these specifications actually change and how that, to me, that feels like it's antithetical to the vision of I write my specification and I never have to change it as the models change. Maybe that's like too far-fetched of a hope. And like, we have to expect being able to change the specification unlike traditional programming.

I think this is where moderation of everything I've said is important. So I think the goal of DSPy is two things. And I think actually, like, I believe they're really aligned with what you guys teach. So the first is nothing is more important in the whole world for the quality of a system. I think you actually guys would agree is then good engineers who know how to iterate and to iterate fast and to iterate on the right things. Like I think when you say evals, you don't necessarily mean like a dataset. You just mean like somebody who sits down, who knows what's going on and who's able to see when failures happen, right? And who can intervene on the right things. So being able to iterate fast is like really powerful. And so being able to have high-level tools in DSPy that's, hey, I wonder if like few-shot examples would help here. I don't know which few-shot examples. I don't know like how they should be crafted exactly, but what, what if I could just apply something that will try those combinations and just tell me, and maybe it doesn't help actually, maybe it helps a lot. It's just that I have these high-level tools. That's the first thing. Second thing.

So the idea is to think of each component in my DSPy program as like a sketch of an idea that I would want to try. And have DSPy is going to fill out the sketch for me. Now, you're not saying that it's going to be like a fixed program forever, but I have to be, as an engineer, be good at figuring out what sketch I want.

Yeah, someone releases a new RL algorithm and I'm like, I don't know, I'm so excited about it. I just run it on my program because I don't need to change the program. There's nothing fundamental about that. But actually, let me make sure I address your main point. I think when we talk about iterating as engineers, I think the most important thing to keep in mind is you start simple and you start with the simplest thing that could possibly work, even if you think it's going to fail, right? So in a DSPY context, you start with a single module, not a lot of like decomposition, counter to popular belief, and you start without any optimization and without having a large massive training dataset because you probably don't have it yet. And you actually just, you do exactly what you guys say, you do the error analysis, you maybe demo with certain users, like you actually just understand where you are, because maybe it's good. Like, we have a lot of users— I think about like at least 40% of our users, they are very comfortable, very happy, and they tell me, Omar, I've never used an optimizer because I don't have a dataset, and I'm like, and we're very happy and we're comfortable. I'm like, great. Like, you don't— there's no pressure. You don't need to use an optimizer if you don't need an optimizer. It's like, I was thinking, um, I'm writing C++ code, or because in a world where Python doesn't exist and the other option is writing assembly, obviously I know analogies are not perfect. And I'm like, I'm not necessarily sure I'm benefiting with GCC or G++ optimizers per my code, but I'm comfortable. I'm happy with it. Sometimes people are like, oh, I have this SIMD and I have all these really complicated loops and I really want fancy optimizations. Like other people, my code is simple. I just like the language. So I think it's like you start simple and you basically like keep adding complexity first. Through the inherent complexity of your system, meaning like you discover that your system actually needs access fundamentally to a source of knowledge that is like impossible to be known by the model because it's literally my emails and it keeps changing and it's private. Model could not have possibly guessed it. So that's in the Gulf of— the first two Gulf, so like actually comprehending the problem and then actually specifying, here's a tool, you couldn't have possibly done a good job without it. And that's what I'm adding first. And I only go ahead and start adding complexity where I'm taking more control into my own hands, where like I'm breaking down steps, not because it's a comprehension or specification limitation, but actually because it makes my language model do better. I only do that when I'm not happy, right? And so the order be, it would have to be like, you start simple, you add complexity that is fundamental to the problem, you run optimizers or other automated techniques to take you the as the first way of handling the Gulf of Generalization, there's no guarantee they'll completely resolve the problem. That's why you're an engineer. That's why you're being paid to handle this problem. So then you're like, all right, I have to step in and have to add some hacks or add some complexity here that is entirely possibly going to be superseded next year when there's GPT-7. And I'm comfortable with that. But I've only gotten there when I was actually required to. So an example of this is, Some of our users write custom adapters. Adapters are the layer in DSPY that takes your signatures, which are these like declarative prompts, high level. They abstract a bunch of stuff and some users actually write their own functions that handle like parsing and formatting and whatnot. And we've seen people like benefit from that and sometimes actually they upstream them. But the fraction of the time that you, that this is the thing that you should like start thinking about from day one or really ever, it's actually small. And my contention is if you're writing raw prompts, which by the way, you can be very productive in writing raw prompts the way you can be productive in many languages. Like this is totally valid if you have the good structure that a course like this teaches. You know, you can be productive in many tools, even in bad tools potentially. You should use the best tool you have. But so what I'm saying is when you're working with raw prompts, you have to commit, right? You have to be like, All right, I'll use JSON because OpenAI today has a structured outputs thing for JSON formats. And so like, I'll use that. But then you realize, all right, like I'm asking the model to generate code and generating code in JSON is actually known to be really hard 'cause the model has to do funny escaping. And so maybe that's not the best thing in the world. So like, all right, I'm gonna switch to XML, which like is a conscious decision you're making. And then you're like, all right, I have to parse this XML in a certain way. So what DSPy does is like for 99% of the use cases, that's just don't think about it. That's not a problem you have to think about. When it comes to the time where like that is really the bottleneck, meaning you've dealt with the complexity that's inherent to the problem, you've ran the best optimizers, you swapped the built-in adapters, you've gotten to the stage where I really have to squeeze 2 percentage points of gains. Now you're like, all right, I'll swap this piece and I'll drop in. I'm sorry, I just, I spoke a whole lot here.

Oh no, it's all good. Maybe let's have— I have a last question for you so you can answer that. Can you talk a little bit about your outer loop? I think that would be very fascinating. You hinted at it a little bit, like you start setting up your DSPy program and do you think of inputs first? And then when you run your DSPy program, like how much effort do you put into writing that initial program? When do you start looking at the outputs? Like, how do you use that to inform what you're going to do next? Like, I think people would find that very valuable, myself included.

So I think it's a question of what are you working on. I think a lot of— so let's actually— let me give you an example that I know is not what people here work on, but just for illustration. Let's say you're working on a benchmark. Like, you're doing a— you're writing a paper, and you're introducing a new method. And as part of introducing a new method, let's say it's not an optimizer. It's like you're solving a system. You're solving a medical problem. And there's already a benchmark for it, but you want to show that modern LLMs can do something cool. In that case, optimization becomes a lot more interesting because from day one, because the dataset exists and the dataset is the accepted ground truth, meaning the gulf of discovery or comprehension is actually a lot less pressing in this particular case. Now, you have to still look at your data. You still have to do error analysis. All these things are important. But like somehow the ground truth is no longer a bunch of stakeholders, it's actually a dataset. And if the dataset is wrong, guess what? That's still the ground truth, right? So that's one extreme. The other extreme is actually the difficult thing in our problem is we have users who are very opinionated. It's like the arena, chatbot arena, for example, or something like that.

Yeah.

I don't know. We really wanna understand these guys, like, or these users. We don't. And like, it's really hard to solicit all of that. And so a huge part of it would just be like, the comprehension aspect. And so I think really that outer loop would just be very similar to what you guys teach. It would just be, what does the stakeholder want? What is the shortest possible program I could write for this? And in a DSPY context, very often we pride ourselves as one line because you could write this very short signature. Just here are your inputs. Here are your outputs. Just do a chain of thought. Let's see what the LLM can do. Use the largest LLM that is affordable for the task. It's only a single step anyway, it's cheap. And then you immediately start seeing how that's going to fall apart because you're like, oh, it doesn't have access to knowledge. I'm missing some important specifications. And I think successful DSPY users start this way with like almost nothing, bare bones application. And then as you iterate, like you realize that your spec is your code because your code is only including things that you've learned the hard way, or at least like in the outer loop, oh, this is actually important. So I immediately take the signature, which is one line, and I make it a class where I can have a docstring where I say like, all right, here are some instructions. And actually, this is a good point. So one thing that users are often confused— actually, this is where more of our more sophisticated users are often confused, like people that have really good taste. They're like, why is the instruction in a docstring in a class? Because I'm not used to putting important content in my, in my code in a docstring. It's a beautiful, it's a beautiful complaint. And I love it because it helps me explain to them only once they get to that stage that the instructions you put in a docstring in a DSPy program, they're not actually instructions for the language model. They're instructions to DSPy. They're actually documentation. You're just documenting what you want. What DSPy actually ends up telling the language model is a slightly different question. And it reserves the right to be like, hey, I know you're asking for the model not to say certain things. If I tell it not to say certain things, it's going to say them because you're reminding it of it. So what I'm going to do instead, in principle— I'm not saying this is happening today. I'm just saying this is part of the— it's consistent with the semantics of the library. Actually, what I will do is I'm not going to tell the model that, but I'll check post-hoc. And if it's mentioned, I'll maybe repeat or I'll resample or something of that nature. So it's a query language where you're talking in English to— to a system that's compiling your software into a runnable language model, which might not actually see exactly the sorts of things that you're saying at inference time. And that's why it's a docstring. It's actually documentation ultimately. Anyway, the point is you expand the signature into a longer one. You start adding more structure in the fields. You start adding the descriptions to the fields. You bring in more sophisticated, I don't know, Pydantic types if needed. You start breaking down the modules. Maybe I switch to an agent, like a React system that I give a bunch of tools. That's usually a good idea because I don't know the right structure. And React agent is a lot more future-proofed compared to manual decomposition because it can actually choose things on my behalf. But then I realized that the system doesn't know how to use the tools in the right order and I have a right order in mind. So then I replaced the one-liner agent with like actually a decomposition, like a workflow where I say, All right, do this and do that, then do this and do that. Sometimes it's the other way around. Sometimes you actually start with the workflow. It depends on the problem. And after I'm happy with the program and it's like things are working reasonably well, if I have a dataset or if, as I iterate on this thing, a dataset sort of emerges from collecting a lot of failures or collecting a lot of tricky edge cases, optimization is a no-brainer. I'll run my optimizers. I'll run multiple optimizers. I'll try different settings. I put my ML engineer hat where training or fine-tuning is now just calling a DSPY optimizer. There are optimizers that will do GRPO. There are optimizers that will, like JEPA, a recent one we released that would rewrite your instructions. There are optimizers that will take your examples, run them through the system a lot of times, and basically look for high-variance examples and then basically take the strand or like the trajectory that worked best. On these kind of tricky examples, and then they make them few-shot examples so the system can see what it does when it succeeds. Very, very similar to reinforcement learning, but in the prompt space. And like, sometimes that works a whole lot better than the thing I wrote. Sometimes I see very marginal improvements and I'm like, all right, that's good to know. If I have to switch to a small model or if I'm curious, which is often the case. I'll be like, I've been using GPT-5, curious if Nano could have worked. I'll just test Nano because I have the dataset at this point. And then I'll be like, all right, it's not working very well. Maybe the optimizers will help a bit more here. Optimizers will often give you more bang for the buck for smaller models just because there's much more headroom. And if you have a larger model, you can teach them. And like, I might end up with a much cheaper system for this. I contrast this with maybe a really common sort of way people are who are really excited to jump into this work. Again, I have a bad string. It's not working. I don't know why it's not working. I don't wanna think about why it's not working. I just want to give you a string and give me a better string. And like, I'll just go ahead and copy that. And like, just to hammer down why that's not a good idea, it's like showing up to a C++ compiler and saying, I have a bit of assembly that is not very fast, and I'll give you my assembly. I want you to give me another assembly that does the same thing, but is a lot faster, but is like a lot more convoluted and weird. and I'll just take it and copy it and paste it into my, as a snippet in my codebase. And the moment you've done that, like you've lost the portability, you've lost the readability. You've, I hear you guys actually talk a lot about this. So people like have weird things in their prompts and like, what is this thing doing here? And how did this even, there are a bunch of emojis and a bunch of things. And I think like often our reaction to that is there's no way that's helping. But actually my reaction to that is, I don't know, maybe it helped. You know what? Maybe it was actually important. But it shouldn't be like the, it shouldn't be that, that like the thing you're managing in the code, because like even if it helps your GPT-4, it's not going to help 5. If it helps 5, it's not going to help Llama. So you are maintaining their own thing.

Yeah. Wow. I like that. I feel like it was hidden throughout your entire workflow, but you're literally building your eval set all the time. Because when you're saying like, oh, I can switch my model and see my result, and you had evals. That's the only way you can do that. And the only way you had evals is because you did the error analysis in the first place to get there. And I think that's so underrated, especially when people just try to over-index on a lot of this, here's my prompt, please fix it for me, give me a better prompt, as you said. It's a lot of great lessons, I think, that folks can take with them. I know we're running super over time, so maybe we'll end it here, wrap it up. I know there's a bunch of questions in the Discord for you as well. Oh, and some people are posting code as well. Okay, so much going on in there, I don't even know. So if you have time, I'm sure people would love it if you could answer their questions, or no rush or pressure on that. But thank you so much for coming and chatting, and thanks to everyone for staying longer than expected.

Thanks so much, Omar.

This is a very interesting conversation. Thank you, guys.

Yeah, of course. All right, bye everyone.