Prompt Engineering vs Context Engineering: What Actually Changed | Mike Taylor, O'Reilly Author

One guy came up to me and said, has nobody told you that prompt engineering is dead? The promise was, uh, as models get smarter, they'll be able to write the prompts themselves. I, I think that's just never ever gonna be true. We have people who are better than most other people at jobs and they still need legal, uh, they still need HR, they still need managers, you know, they still need, uh, employee handbooks and all of these things are prompts.

But you're applying the same skill.

I actually use my managerial skills more with AI than I use, uh, my data science.

Welcome back to Two Sided Act. Today, today I'm talking with Mike Taylor. He's an O'Reilly author, columnist at Avery, and the CEO of a product called Ask Rally. They're the company who's building AI focus groups with synthetic personas. So, Mike, thanks for joining.

Yeah, thanks for having me.

How did you end up writing a technical prompt engineering book? And now you're writing, I know you're writing another book about context engineering with O'Reilly. How did that happen?

A lot of brute force and ignorance. So, yeah, so I came from an economics background. I did a master's in economics. So, I wasn't like afraid of technical stuff, I would say.

You were?

I wasn't. Yeah, I wasn't.

Who wasn't?

In my first job, I was like, what are these guys who build a lot of Excel scripts? You know, so, I was like semi-technical, I guess. But, but yeah, then, you know, built this marketing agency, grew it to 50 people, spent about 50% of my time managing the tech team because we automated a lot of stuff and did a lot of data science as well. It was like a growth marketing agency. So the assumption was that like we'd know how to run A/B tests. We'd know how to calculate statistical significance. We know how to install tracking and the website, you know, things like that. So got pretty technical that way. And I was usually like the guy who they would bring in to say like, hey Mike, how does this work? Or the developers tell me this, I don't really understand what they're saying. So I wanted to learn how to code really to, you know, be able to talk to the developers more. And then when I left in 2020, after like, you know, maybe 5 years of being semi-technical running this agency, I just started coding full-time and did like a bootcamp. I think it was DataQuest was the one. Like self-paced, kind of burned through that. And then I started just freelancing as a developer and nobody knew that I didn't have any experience and all they cared about was whether the code worked. So it was, it was fine, you know, so I just, just kept doing that. And 4 years later, because I'd gotten access to GPT-3 in 2020 when I was first messing around with this stuff and was blown away by it, like I suddenly like was like, one of the few people, I guess, who had multiple years experience of prompt engineering when, when, when O'Reilly came to write a book, wanted to commission a book on the topic. So, so I was lucky enough to get into that position and then now doing a second book with them on context engineering with DSPy.

Is the essence of prompt engineering basically the ability to talk to AI?

No, I would say that you want to separate prompt engineering from just prompting, because sometimes it gets stuck in the same bucket. Actually, a lot of times it gets stuck in the same bucket. And I think that if you're going to call it prompt engineering, there should be some engineering. So, you know, should be testing, A/B testing, right? You should be, you know, figuring it out if it works at scale. You should be designing workflows. And now the sort of things that people call context engineering, I was doing as a prompt engineer, like You know, the difference is the models, you know, stop working today after you fill like the context window at 200K tokens. You know, we only had 2,000 tokens or 200 tokens depending on which model you're using to play with. So we're already thinking a lot about how do you get all this information outside and then only put in the right amount of information to do the task. So, and we were thinking a lot about how do you formally evaluate these things. So I would say getting good at using AI, one way to get good at using AI is to, you know, formally test and define inputs and outputs of each workflow. But I would say there are people who are really good at prompting that don't know how to do prompt engineering and vice versa, actually. There are people who are very good at prompt engineering who are terrible at writing prompts. They just, they just kind of engineer the system to write the prompt for them. Because like, if you're just prompting for yourself, like most of the prompts I do, I don't really, I don't think about it. Like I just say, Actually, my most popular prompt probably is just like, fix this, like, and then a screenshot. Oh yeah. Yeah. So, so yeah, I would, I would say like, I'm actually very lazy at prompting, but like very diligent at prompt engineering. Is it done yet? I remember I went to a conference in 2023. I think, I think it was 2023. It was the AI engineering conference in San Francisco, the first one. And, and it was really interesting actually, but, but one guy came up to me and I had prompt engineer on my badge and he said, has nobody told you that prompt engineering is dead? And he was like, you need to look at DSPy and, uh, and, uh, and like, uh, what, what they're doing. And I think it was a paper out at the minute at that time called DSP. Like that was the initial kind of paper I think Omar was working on. Anyway, so the, the promise was as models get smarter, they'll be able to write the prompts themselves, you know, or they'll be able to maybe don't even need prompts anymore because they'll be able to guess the right context and I think that's just never ever going to be true. You know, we have people who are like, you know, if the definition of AGI is you can do every job as good as a person can do the job, like we have people who are better than most other people at jobs and they still need legal, they still need HR, they still need managers, you know, they still need employee handbooks and all of these things are prompts, right? So I would say we might not call it prompt engineering, but it will still matter no matter what happens, no matter how smart the models get, it will still really matter as to, like, can you communicate in a reliable way what type of outputs you want from this model that you're interacting with?

I can really relate. I feel like talking to my kids is one way I'm practicing prompt engineering. It was literally prompting a lot, but I realized I really have to you know, do different variations and try different things because things happen again and again. And like, what is the result? I want different results. So I really want to put it some— put some— maybe I should, you know, leverage your book about that, about parenting.

There you go. Yeah, it's a parenting book in disguise. And I have two young kids as well, a 4-year-old and a 6-year-old. So maybe, maybe some of that seeped in. But the problem is it's very hard to A/B test kids because they, they, uh, they realize what you're doing and they revolt, you know. Whereas AI is much easier to A/B test.

Yeah, they're too smart, right? They're too smart. And I think, I think it's because they're accepting more data points about mommy, right? It's not just how I talk, it's like my facial expression, you know, how I'm acting, like what happened during that day. And I, I think they can tell, they can tell that I'm pretending I'm trying out a new parenting technique on them. And they're like, you don't sound like you, mom.

Yeah. Yeah. What's the deal here?

Yeah.

Right, right, right. Yeah. And that's true for, actually, it's a funny analogy because that's like my big thesis as well that because I used to be a manager of people, I managed 50 people at the agency. I actually use my managerial skills more with AI than I use my data science skills because most of the time it's about, did I provide a clear brief? Like, did I get it to plan first and accept the plan before I let it go ahead. You know, did I divide the labor up in between different workers or whatever so that, you know, so that like one of the workers doesn't get too confused or overloaded? And it's all the management theory, but like actually applied now because you, like I said, you can't A/B test people very easily, you know? Right, right.

Good point. Yeah. It's a lot easier with AI, right? But you're applying the same skill.

Yeah, exactly. Yeah. And I think that actually, maybe if it's not called prompt engineering in the future, what it might be is just part of the business school, you know.

You have so much experience in prompt engineering. If you were to give your best practitioner's tip for people doing prompt engineering, what are they?

I would say that like almost everything comes down to, do you have enough good examples of the task that you're trying to solve? Like everything else is incidental to that because everything else is like style and formatting and that can all be inferred from examples. So like, and I think that's also the thing that people underinvest in. So even the very experienced machine learning engineers, like, because a lot of the time the tasks that they might have collected examples for, built datasets for were like classification tasks. Where you're just like, one or zero, you know, is it this label or not? Yeah, yeah, exactly. Hot dog or not hot dog, you know, then, yeah, then they're usually pretty uncomfortable with like, okay, we need to automate blog posting. So let's find 1,000 good examples of blog posts. And it's like, I don't know, I might have 4 or 5. Yeah. So whenever I've started working with a new client, that's like number one like we create basically a swipe file of, you know, here are the things that you really like, maybe even collect a file of like, here are all the things that you really dislike. And then because AI is so good at pattern recognition, it's much easier for it to explain what's different about those two groups. Then like, you don't actually need to worry that much about writing the prompt if you have a good set of examples.

Because humans may not be so good at describing the nuances, but AI can actually spot the pattern. So just providing the examples.

Yeah. And that's one of my favorite, like nerdy, like underexplored uses of DSPy is because it's a prompt optimization. Well, I mean, it has this capability of prompt optimization. We can talk a little bit more about it. But when you run the, like, say the JIPTA optimizer is like the new, the new hot one that everyone's running. You get at the end of that process, a plain English prompt. That has captured all of those patterns. It's basically like a document of, you know, if someone had spent a lot of time really understanding the examples you gave it, like what patterns would they find? And it's fascinating. Like I even tried to clone myself. This is like part of the work for Rally. And then what I got at the end was a prompt of like exactly how I talk, what are my mannerisms, you know, like what things do I like to talk about, you know, and it's very revealing when you when you look at the prompt.

You know, I have a little bit imposter here. What's the difference between passing along a bunch of examples versus passing along the descriptions?

Yeah, it's a really good question. And I would say it's better to start with the examples if you can. But description is a shorthand, right? Like it's a generalization. So if I, if I, if I, if I give you like, I don't know, 100 pictures of cats, 100 pictures of dogs. Like, you know, and you run that through DSPy or whatever, it's going to figure out, like, okay, you can classify pretty easily with a few examples, which one's a cat, which one's a dog. But, you know, you could also just describe what, like, what are the features of a cat? What are the features of a dog? What are the differences between them? And it might be able to get there with a smaller prompt. So a lot of prompt engineering, and particularly context engineering, which is like engineering the overall system, is to, like, see what you can do with less. Because part of the reason why companies would hire someone like me, uh, when I was doing this freelance was, uh, they wanted to decrease costs. Like, they had it kind of working, but it wasn't very reliable and it cost too much because they had to use, you know, the latest model, like the Frontier model, which costs, you know, 10, 30 times more than an open source model. Uh, so a lot of the work is actually just distilling that down and say, can we arrive at a description that still gets good enough results without having to include this huge prompt, which increases the cost because you're paying by token.

Does the Deep Optimizer overfit? You know, it sometimes can overfit, right?

Yeah. So it depends on which one you use, but it uses like an evolutionary algorithm where it does like mini batches and we can get like pretty deep into details. But, but yeah, broadly speaking, you think of it as it has, it has like a holdout group, like it has some, some examples it hasn't seen. And then even from the examples you give it, it will take off a small batch of those examples, like 3 examples. And it will just test the prompt on that batch. And if it works, then it will test it on the wider, wider case. And then, and so when the act, when it does the accuracy, it will give you the accuracy on like the test case. So the examples is not learning from. But I also keep another holdback, which I call like the development set, which I think is pretty standard practice in machine learning. And you have this other set that you didn't even give to the optimizer. So then you can, you get the compiled program at the end and then you can run it yourself and just double check that, like, even for things it hasn't seen, it's still working.

Yeah, this is a traditional machine learning skill, like, right? Completely out-of-time, out-of-sample test set that you're just doing like when your dev set is done, then you're trying it on that holdout example.

Exactly. And actually, if you come from machine learning, traditional machine learning, like you'll actually love this program. I didn't, right? Like I had done, you know, machine learning basic stuff, right? Like I had done data science, but I didn't come from an academic background. And if you read like the documentation for DSPy, you realize that it's very academic, right? Like, and very intimidating actually to me initially. And it took me a long time of like just Googling stuff and asking ChatGPT, uh, what does this mean? What does that mean? Uh, to like finally get my head around it. But once it, once you understand it, it's great. But I think it's actually built for people from the machine learning, uh, discipline.

How many examples do I need?

Yeah, I would say that if you have less than 10 examples, you will get most of your gains literally just by like adding a few of those examples to the prompt. And you probably don't even need to like do optimization. If you, I would say if you have more than 10, a good prompt optimizer like Jeeper, which optimizes the actual prompt instructions. There are other prompt optimizers that optimize which examples to include in the prompt. Like you might have 100 and it chooses the best 5. But yeah, for something like Jeeper, it can do a pretty good job with at least 10 examples. But yeah, I would say for something that's like mission critical, you're using in production, you're trying to get it down from like a frontier model to an open source model, then I would say like 200 is what I try to aim for. And you don't have to start with that, right? Like, I think the best thing is just to get it in production or get it in beta, you know, get people using it and you can quickly get 200 examples that way. You know, like, and I see it as a graduated thing. So at first you're just vibing, you know, like, do I think the results look good? And then you're sharing it with the team. Do they think the results look good? And then you, even when you've done that, like if you're just logging what people asked in your chat window, or if you're just logging like what examples it generated, now you have like probably like at least 10 to 50 already that you can use as an optimization. And I don't think it's like a one and done process. I think you can actually just have it like you running a regular optimization every week and it, you know, it just captures the user preferences slightly better this time.

This really reminds me that AI is a data problem. This is a data curation problem.

Yeah, it actually really helps, I think, to look at an LLM as a search problem rather than a generative problem, because they don't actually generalize very well outside of the dataset. It's just the dataset is very large, right? Like the dataset is the whole internet, right?

Right.

And if it hasn't appeared on the internet, they tend to hallucinate really badly. Like they tend to, you know, make mistakes and stuff. Yeah, exactly. So your job, I think, as anyone who interacts with that, making that product better or making that workflow better, is really just to kind of think about like, is the task in distribution? Is it common in the training data? And if not, like, how do I make that more common for the model by like either putting that into the prompt or fine-tuning the weights of the model? And I don't have a lot of experience with fine-tuning, just because usually just optimizing the prompt works straight away. I never really have to reach for fine-tuning. But yeah, like, you know, what can you give it to ground it in that task? And I think then it can kind of do the rest.

I'm curious if you can quantify like my ROI in terms of how, you know, how as a developer, I'm developing something like this, how much incremental gain I can get.

I would say that. Even straight away, like even with like 10 examples, I've seen gains as high as like 50-60%. But it massively depends on how complex the problem is. Like if there are a lot of edge cases, then the chances of that one edge case being only in your training data and not in your test set or vice versa, that imbalance is actually what tends to hurt the accuracy that you report. So I would say if you have like a perfectly balanced dataset, so you have, you know, in your holdout, like, so maybe there are only like 3 bad things that could happen and you have one example of each in training, one example of each in test, then, then, then I would say you're, you're probably going to get like maybe at least like 10 to 20% accuracy, maybe as much as like 50, 60%. I have seen cases where GPa has doubled accuracy from a low, low bar, you know, But it also depends on like what you started with. Like if you, if you got an LLM to write your first prompt in the first place based on an example, I would say the gains would be a little bit harder to get. If you had a prompt engineer write your prompt, then the gains are a little bit harder to get. But, uh, like I would say I can still beat the optimizer, like if I work really hard for 2 days, right? Like I still, I still have prompts that like I haven't been able to beat. I now very rarely like spend 2 days on a prompt. I just, give it to JIIPA and say like, you do it. So, so I think that that's actually, it's not just about overall accuracy. It's also just about like workflow management and like, what do you want to spend your time on? Like, do you really, like most of the prompts that I have in my system are like throwaway prompts. I don't care that much about. And then I don't even need to optimize because it kind of just works every time, first time. So I'd say like, you don't do optimization for the sake of it. But if the prompt is like very important, it's costing you a lot of money, or it's really unreliable, then you need to sink more time into it and more money as well. Because this costs money, right? Like when you— actually, I spent a couple of grand on an optimization process for a client. And, you know, like the credit card that I had on that OpenAI account was like maxed out. And I was like, oh. God, so I had to call the client and say, I'm really sorry, but I need to spend another couple of grand and can I have your credit card? Uh, yeah. Uh, but that was like for, that was like for a huge personas project of like 5,000 personas. We were building the personas and then training them to be more accurate. And so it was like a lot, uh, you know, so, so you, you shouldn't cost you that much for DSPy. I would say like typically it's, it's like under a dollar for an optimization run. Maybe a couple of bucks if it's a few hundred examples. But it's also like a case, like usually when people go for prompt optimization is because it's costing them a lot of money not having that prompt optimized. So they're willing to try anything. It's like, you know, OpenAI is like the biggest, you know, cost in my company right now. It's more than I pay my own salary. So like, can we like get off of this into open source models?

And then the next step, people want like open source models, free models, smaller models, and can you fine tune it for the performance, right?

Yeah. And I would say this is like much more accessible. It's much cheaper than fine tuning. It requires much less technical talent. Although I would say you can actually fine tune through DSPy as well. And that might make it a little bit more accessible for people who haven't tried that yet.

Besides the two books you're you published and you're working on, you're also building Ask Rally, right? Tell me about it. Tell all of us about it.

Yeah, cool. So one of the things we never did enough of was market research at Ladder, my agency, because we came from a performance marketing background. So we were like, oh, we could just A/B test everything, you know, and that's not always true. And it's very expensive actually to run A/B tests. And, you know, a lot of ideas that like, you know, you could have vetted them basically and either come up with better ideas or just avoided doing that bad idea in the first place. So I developed a really strong belief in market research through that process. And then when I saw my prompts that like every prompt was starting with, you know, as this person, you're going to do this task, as that person, you're going to do that task. Like it was a lot of role play and people were using AI for creative role-playing games and all sorts of things. It made me realize like, hey, why don't we just get to roleplay as a customer and see if it's useful? So it was useful. And that was what kind of became Rally. And there's a lot of other people in the synthetic research space, you know, and I've actually talked to a bunch of them. They're really like good. And the market is still like very nascent, you know, still trying to figure out where the product market fit is, I think, for most people. But for us, what we settled on is more of like a self-service approach where like probably the lowest cost out of anyone because a lot of the others work mostly with enterprise clients. And, you know, we, you can set up a credit card and start using us, right? We tried to keep it as close as possible to using ChatGPT because that is what like the alternative is for most of our customers. Like there are people simulating their customers in, in ChatGPT. The benefit of using Rally is that it's like chatting to many GPTs at once because you get like the whole focus group and the personas are stored. So you can keep coming back to those personas. We also did some pretty interesting process around, you know, as you can imagine, like optimizing the prompts, like optimizing the actual personas themselves so that they're, you know, actually based on real human data and like real human interviews. And, and yeah, we can, we can go pretty deep into that, but that was the gist of it. Yeah.

Who's the product for?

Yeah, good question. We're still trying to figure that out. We haven't found product market fit. I'll be like perfectly honest. We have like, you know, say like 50 customers right now actively using it as well as doing a few other like bigger projects for large agencies. Yeah, I would say the most popular customer profile is like a small business owner or someone who works in marketing in a small business where they're, or actually in some cases product. Where they have a lot of decisions to make. They don't have a lot of budget to do market research, but they understand the value and they also understand the value of AI. Like actually, that's the biggest determinant for us is like if someone's using ChatGPT or Claude all day, then they completely get it. And, uh, but then there's a lot of people in the market research community that hate AI and see it as a threat and, uh, see it as, uh, it can never do what we do. And so, so I would say like that. That we initially thought the market researchers would love it because it's just extra capability, but yeah, there's a lot of like bad juju.

Did you do market research using Rally for your own product?

I wouldn't say you use it on large strategic decisions because I think that they actually have to be made more based on your personality and experience. But for example, we, you know, run every ad through Rally. Like I publish my LinkedIn posts through Rally. Um, yeah, when I launch a new landing page, I ask, uh, you know, what do you think of this versus this? So, uh, because you can do images as well. So, um, yeah, I use it on, um, I would say it's like really good on like the middle of the road decisions where you don't have like a very strong opinion and, uh, it's not like majorly strategic, but it would be good to get a second opinion.

Are you saying that I can simulate that first with Rally?

Yeah, you can simulate that first. I don't think you should skip that step, by the way, but But what you can do is like find the right 10 and rule out some of your worst ideas first. So you can bounce your ideas back and forth. I would say one benefit of Rally over ChatGPT, and the reason why I— it was initially a script that I was running for myself before I turned it into a product, is that ChatGPT is very sycophantic. It will always tell you ideas are great. You know, it's like your mother, you know, it's like, yeah, awesome job, sweetie, you know, and then you take it out to market and people rip you apart. Part and you're like, oh man. Yeah. So, so I would say it's really good at helping you avoid the worst ideas. It's really good at making you customer focused because a lot of decisions you make without market research, it would have been avoidable to, uh, that mistake would have been avoidable if you had just like heard a little bit from the customer. So, uh, it's, it's too easy actually to, uh, to just plow ahead because it feels right to you. Um, and then it's not that you don't know that the customer doesn't like this or does like this. It's just that like, you need that reminder actually sometimes. And you have that knowledge in your brain somewhere, but like, it kind of brings it back to the front. And you know, there's like this elevator problem with CEOs, where like, the last, like the new strategic direction is always what the last person talks to them in the elevator about, you know, so whoever gets in the elevator with the CEO, like shapes the direction of the company that day. Yeah. And it's kind of like that. Like, I think it helps you avoid that basically. Absolutely. It kind of refocuses you on like, okay, well, what are my actual like customers potentially going to say here?

How do you actually create these customer personas for the focus group?

Yeah, good question. We're trying to actually make it so you don't need to. So we have our own panel that we've done real customer interviews with 300 people. We have used DSPy to calibrate those personas so that the responses match, you know, to a high degree, the same style, the same substance that the real person, you know, delivered in the, in the real interview. And we found really good results from that. So you can basically just pull down from that group. But there are also other ways to create personas.

Are they consumers?

Yeah, this is like more of a consumer audience. I would say it's like general US demographics. Obviously there are like some business owners in there, some, you know, but it just depends on what type of product you're launching. I'd say for B2B use cases or international use cases, typically what you want to do is you can go and do this process of like interviewing some of your customers and then uploading them into our platform. And that tends to work best because this is grounded in a real person, is grounded in a transcript that came from a real interview. And therefore the results are going to be much closer to real life. Yeah, we also have the ability to generate AI personas purely through AI, like with no grounding. But I would say that then you're introducing some bias in that, I mean, bias is a funny word because we actually want the personas to be biased, but just like biased in the same ways that humans are, you know, so that it's realistic. But like, I'll give you an example. We ran a fake election. This is like before the models knew that Trump won the last election and 90% of the personas voted for Kamala Harris, you know, so like, that's obviously not what happened in real life. You know, a lot of people tell me that's what should have happened, right? I wouldn't step in, but, but like, that's, uh, that was the interesting thing is it's like very biased in a way, like to one side, um, in a way that like the real people in, in, in America were not right. They, they, it was more 50-50, right? So, so yeah, that I would say like when you, because the AI models are like trained to be very politically correct and they're trained to be like very helpful and lovely to work with, then they tend to be biased in those directions. They tend to say like, okay, you know, they'll vote for the more polite candidate, right? They'll vote for, you know, they'll watch La La Land instead of Transformers, you know, even though Transformers makes more at the box office in real life, you know, it's not just politics, it's like everything, it seeps through.

Do you think these synthetic personas of a real individual can actually predict what people will do or how they really think?

I think we have free will, so I hope not, you know, but I would say that it can predict what would be plausible for the person to do. And the distinction I'm making there is I cloned myself and I asked it, what's an embarrassing purchase you made? And it said, at 2 AM, I was browsing the internet and I found a lightsaber, build your own lightsaber kit. And I bought it. And, uh, as soon as my wife saw it, she thought I was an idiot. And, you know, the kids loved it, whatever. And I was, and I was reading it, I was thinking like, that didn't really happen in real life, but like, man, like, that's exactly me. Like, that's exactly the sort of— I've done, I've made purchases similar to that. Like, I've done, uh, you know, it's exactly how my wife would react. And, you know, so it's, it's, um, it is not something that really happened, right? Like, it didn't predict what really happened. Uh, but it did, uh, it was consistent with my personality. It was consistent with my interests. I think that that's actually like, as, as far as we can get, right? Like, I don't think this is like, you know, Minority Report, like predicting future actions and, you know, making decisions based off that, right? Like, um, I think this is more, you want to know what's plausible and, uh, you want to know, uh, what, what might people say about this idea that I have. And the reason why that's useful is so you can anticipate their objections, is so you can, you know, understand like, what would make it resonate better with them? Like, what would they like more? Or like, how would you, how can you serve them better? And I don't think you actually have to predict their actions to do that, especially at scale, right? Like, you might not be able to predict that I am going to buy it, but maybe if I'm in a group of 1,000, you could get like pretty close, like 80, 90% accuracy in predicting how many people out of that 1,000 will buy, you know? Yeah. Yeah. Do you like the chicken or do you like the— no, it's like, it's, it's like that. Like, so I think that the way we survey is all wrong in the age of AI. And, and, um, and you can think of, uh, these semi-structured interviews as a way of making it comfortable for the person, the human who's answering these questions. Uh, so they give you a lot of information that you can then use to triangulate what their preferences would be like. If you were going to try a new idea out for size.

And are you using fine-tuning or training on each one of the personas or it's prompt engineering?

Yeah.

Yeah.

It's, well, it's, it's automated prompt engineering, but yeah, it's, yeah, we actually haven't found good results with fine-tuning. And mostly because we use, you know, some of the frontier models where it's like more way, way more expensive and like, it doesn't lead to that, that big of an uplift. So, so we, we have two parts to the process. One is we create an LLM judge, uh, that, uh, that's the job of that judge is to tell whether it's looking at a real response or an AI response. And, uh, and then, uh, so then we, uh, when that judge is really good at telling the difference, then we use that judge to train the prompt, uh, for the persona until the persona can trick the judge. And then once the persona is consistently tricking the judge, then we can go back and use that persona to train the judge to get better. Because in each case we know which response came from the persona. So we can, we can weight that in the, in the reward function. It's like a classic evaluator optimizer pattern. But like, I remember like GANs back in the day before, before LLMs and yeah, like before image models, diffusion models, like these generative adversarial networks. I always like intuitively like that, that approach. So it was like one of the first things we tried. When we were trying to solve that problem.

Like two persons debating with each other, right? And then you in turn improve both of them.

Yeah, exactly.

There was a Google or Stanford study that talking about digital twins can replicate human thinking with 85% accuracy. What does that 80% accurate actually mean in practice?

Yeah, it's a really good thing to ask because there are a lot of these papers and I remember there's one recently with PyMC Labs and Colgate, I think, also found similar levels and they're all true, right? Like we've seen the same thing with some studies as well. Like we did, we predicted gamer preferences. Like we had, we had some real, real life game design studies that we replicated that a friend of mine shared with us. And yeah, it's like, I think if there are not too many options to choose from and if the study was done in a survey format. It's actually fairly straightforward to get 80%+ accuracy. It's basically impossible to get more than 90%, by the way, because if you gave someone the same survey twice, then they won't even agree with 90%. They'll be 10% different the next time that you survey them. So I think that's actually the ceiling and people claim more than 90% maybe just not really being truthful with you or they're discounting in some way. The, I would say what's a lot harder is replicating the experimental environment that the original research was done in. Uh, so one good example I have is like a lot of people came to us and said, uh, I ran this test on Facebook ads. Can you tell me which one won? And then I'll trust your product. And the problem is that like, if you ran that test on Facebook ads again, you know, just so happens that today's Valentine's Day instead of, you know, a regular day, then you're, you know, the results you get on Facebook are going to be so different the next time you run that test. And we can't control for all those variables, right? Like, you know, so I would say like, it's, it's, it's going to be directionally correct. Like, it's usually pretty good at picking the winner. And yeah, it's still very useful. And, and the nice thing about this versus an A/B test is that when you run an A/B test on Facebook ads, you don't get to know why they clicked or didn't click, whereas with Rally you do. So I would say, again, it's not like a replacement for those the tests that you're already running. It's not a replacement for the market research you're already doing. You can think of it as a way to eliminate the worst ideas and improve the best ideas, and then go and run further testing, right? So that's kind of the way I would think about it. And don't expect it— it doesn't have to perfectly replicate an existing real-world study in order to be useful.

I get this a lot on password resetting questions. They keep asking me, what's your favorite movie? I'm like, you asked me last year and asked me different, right? Why are you asking me this stupid question on password reset. I can never remember my future, right? But this is the same thing in a sense that humans are not consistent by themselves. So we are— if we ourselves comparing with ourselves is 15% off, and then you're saying the persona can get to 85%, then you're pretty much reached the ceiling. So, so that kind of justifies approach, right?

I think so. How I would behave I think so. And again, I think that everyone thinks about it as, will this tell me whether my product will be successful? I'm like, no, sorry, there's nothing in the world that is ever going to tell you that because it's a dynamic system, right? If you're lucky that, like, I don't know, The Rock sees your product and then tweets about it or something, then maybe it takes off in this universe and maybe in a thousand other universes it wouldn't have, you know, I just, you just don't know, right? There was actually a paper, really cool paper that looked at this for music, and they found that if you— they tested 12 different groups where their job was to listen to the music, and they were all songs that people had never really heard. They were very low rated on Spotify. And they found that some of the worst songs became top of the charts in some of the universes just because they were lucky enough to get a couple of likes early on, and then they rose up the leaderboard, and then other people jumped on that bandwagon and the social proof, you know. So, so I just think it's, it's like, if you're looking for something that's going to tell you whether you're going to make a million dollars with your product idea, then like, uh, you know, that I think that's the wrong way to think about it. I think it's more about how can I get more lucky, right? Like, how can I avoid doing obviously wrong things? And, and, uh, and Rally will help make that obvious. Or, uh, you know, how can I improve the things that I'm already doing? Like, how can I just, um, tweak them to be a little bit better.

I don't even think that with real human being market research, you can answer that million-dollar question either. Exactly.

Yeah, they would be paying a lot more money for focus groups than, you know, it wouldn't be a $20 Amazon gift card if it could really predict the future, you know, it would be like a, you know, $1,000 Amazon gift card.

Do you think people actually pay for these AI-powered market research tools? Is there a real market here?

There is. I know that there are like people in the space making like tens of millions in revenue, but it's still like very early. And I don't know like how many, how many of, how much of the revenue is like pilot projects that all AI companies get versus like, you know, like we're going to be using this forever. Yeah. Yeah. Yeah. So, so like that, that is unclear to me. I'm not like in a good position to understand that because our impact on the market is so tiny, you know, relative to some of the others. But, uh, but yeah, like I actually, my hypothesis, the reason I got into this is that, uh, once the content becomes more AI generated, so, you know, like once you're generating ads with AI or once you're generating products with AI, like, you know, I can vibe code something in a day now, you know, half a day, and it's pretty good. But the reason I don't is actually I don't want to like every single day send out a new product to my friends or my colleagues and say, I did this today. And they're like, wait, I thought you were working on this problem. I'm like, no, no, I already solved that. I'm doing this today. You know, and, um, and they're like, why don't you just focus? You know, I think attention is limited. And, uh, and I think that like, there's going to create this huge demand for like, how do I validate which thing to work on? Right. So like, if you can. You know, if it used to take a whole year to launch a product and therefore, you know, every time you launch, like people would pay attention. Now nobody's going to pay attention when you launch. So you maybe need to use a service like Rally or one of our competitors to, to like validate which idea to launch this year. You know, maybe you, maybe you build 500 things, but only one of them sees the light of day, you know.

Why, why are people paying you now? Why do people pay you now?

Yeah, right now I would say, uh, The people who really get it, who like have a lot of ideas that they want to test, and they've messed around with ChatGPT enough to know that it's not fit for this purpose. It's a pretty small Venn diagram, right? But it's growing and, you know, and I think like that's, that's kind of the target base at the minute. But, but yeah, like I think as, as more organizations adopt vibe coding tools like Replit or Lovable or even like Claude Code or Cowork. I think that that's going to create a lot of demand for like, okay, if we're going to be vibe coding, you know, how do we need to vibe research or something?

I think the vibe coding tools really enable everybody to become a core, a builder. Tell me a little bit about your pricing versus like a real focus group, human-based one.

Yeah, so we have two plans, $20 a month and $100 a month. So, you know, to put that in perspective, to run a focus group might cost you like $5,000 to $10,000, something like that. I think you don't have to do that, do it that formally. I think you can do a decent job at a focus group just by inviting a few people in your target demographic, like tracking them down, cold emailing them something, and then offering gift cards. I think you could do it for a few hundred if you needed to, but actually it's not necessarily even about the pricing. It's about the time it takes., right? Because, uh, to do that, right, like you're now going to spend a whole week like putting that focus group together. And then once it's done, you can't ask them anything else, you know? Uh, so, so I would say that, uh, yeah, this is more about like you get instant feedback in the same time that it takes to ask ChatGPT. Uh, but it's not going to be sycophantic. It's not, you know, it's actually going to tell you if it thinks the idea is not good. And it's, uh, each response is unique, uh, to that person, to that demographic, to that profile. So you get a wider range of responses rather than just like the average of ChatGPT response to me.

What's the typical size of a focus group? Like how big of a group do I need?

Yeah, we, we've, we start, we, we do 5, uh, as the default, but, uh, we can scale it up to like 100. It just depends. I would say if you're doing ideation, like if it's really important to you, um, if each individual objection is very important to you, then you want to look at the individual responses. And so we keep it at 5, but if you're more into validation, like you're about to launch. And you're just like, okay, we've got these 3 ads and we want to decide which one to run. Then, uh, cause we can only run one, uh, then, then like, then you want to scale it up and do like 100, maybe 200. And then get that because then you can tally the votes.

How's it different from using Rally?

Yeah, exactly. So, uh, so you could, and I used to do this before I built Rally was like, you could do, I used to call this personas of thought, my prompting technique. You can just literally just ask like, hey, come up with 5 different personas of people who would buy this product. And then tell me what each one individually would think. And then summarize their responses at the end. And what you would get is something like, you know, tangential, like similar to RALL-E, but it'll still be a bit too sycophantic. It will still not, it will lack some of the diversity and And also, because they can all, like, obviously not 5 personas really in there, but because they're all in the same thread, I call this the vegan problem. Like, if one of the personas is vegan, then the other personas tend to be more vegan. Like, they say that they wouldn't eat meat or something. And I'm like, you're not even supposed to be vegan. But because they're in the same thread, like, they color each other's responses.

This has to be context engineering, right?

But Rally, it's not that problem. Exactly, it's context engineering. Yeah, with Rally, it's all individual. So, we're literally sending off 5 different prompts to 5 different, you know, models essentially. So like we get, you know, we get, we get back like these isolated responses and then we summarize at the end.

So you are really creating the individual personas.

Yeah. And they're stored. You have an audiences tab where like those personas live and you can, you know, start a new session with the same personas or, you know, wipe that session, start another one. So yeah, that's like really useful for people who know about context management. And, uh, context rot is exactly. So, so, um, yeah, that's, uh, but I would say, you know, if you don't have the budget for Rally and you're already paying for ChatGPT, I still think it's valuable to—

it's the same price.

Well, I mean, the thing is like people are already paying for chat, you know, they don't want to add more SaaS. I don't know. But like, yeah, I still think they should buy Rally.

I grew up, I guess a lot of us grew up, we were told to sync long-term, but technology and AI is moving so fast now. How do you plan when the ground keeps shifting?

Yeah, it's a really good question. And it's something I think about a lot because to some degree it feels like if you don't keep up, then you might not be able to keep up, you know, because if someone is like, I think about this, I don't know if you've heard of the permanent underclass theory. My abbreviation of it is like, uh, so there's 3 people. One is using free version of ChatGPT. Or maybe not using AI at all. Another one is using like Claude Code Max plan and like vibe coding stuff. And then, uh, there's someone else I was thinking of, Jeffrey Emmanuel, uh, Doodlestein, I think his name is on Twitter that like he's spending like $4,600 a month on, he's got like 22 Claude Max accounts that he's continually running and his whole screen is like, you know, I think he used to be a trader, right? So like he's, that's probably why he already knows how to context switch so much. But like, for me, that's crazy. But, but like, uh, you know, if he's basically getting like maybe 1,000 times more usage of AI than, uh, you know, someone who's using the free version. So does, uh, he run away and with it, like, and, uh, and then like right now the, it's $200 a month plan. Like Sam Altman has said, there's going to be a $2,000 a month plan and a $20,000 a month plan, uh, which is like PhD in your pocket. So like if your competitors are using the $20,000 a month plan, because they're so hyped on AI and you still think ChatGPT is not good enough and you haven't bothered to use it, then will they just like run away with it? And then, you know, then, then the $200,000 a month plan comes out and you don't even make enough money to afford that. So that, that's the, that's the scary fast takeoff thing. I actually don't think that that's going to happen for a few reasons. Uh, and, and I think there's also some stuff you can do. To position yourself well. So one is, I don't think it's going to happen because we have open source, we have like the price of these models are coming down 90%. So yes, a PhD in your pocket might cost 20 grand today. But is that really going to matter when like next year, it only costs 2 grand and then the year after it only costs 200? You know, like at some point the value of a PhD is going to collapse, right? If that's, if that, like, and this is also assuming that they even, you know, get to that level. Like, we don't know if LLMs will keep scaling, but let's assume they do. So I think that the price collapses quite quickly. I think that it's very hard to, to like change the physical world, uh, with software. So, you know, there's all of the physical jobs, uh, that require a human are going to be way higher in value, uh, because there'll be some people earning a lot of money that like inflates the value of everything else. And then I also think that there's going to be this comparative advantage thing, right? Like, if there really is a model that's worth $20,000 a month, then it's not going to— they're not going to want it to run it on the tasks that me or you are doing, right? Like, they want to run it on frontier science. Like, the government would be using it. And, you know, Fortune 500 companies might use it. But, but like, it's, it's actually going to be cheaper for a company to hire me to you know, do some data science because, you know, like, it's like, why would they want to retask that model away from foundational science? You know, they wouldn't, like, because, you know, like, who else would be doing it? You know, so, so I think there is this competitive advantage, like, at some point, I think intelligence just costs. I do think, like, humans are pretty efficient. Yeah. Yeah. Like, human intelligence is, I think, actually, like, a very good deal. For how much it costs to feed us, you know, like for how much it costs to feed us, like we are AGI, right? Like, you know, like you get AGI for, like, you know, what's minimum wage, right? So, so yeah, I actually, I'm an optimist. I totally see the doom loop that we could go into, but I don't think we're going to get there.

We're trending that direction. Yeah.

Yeah. And then, and then I also think that to position yourself well, I think you should try and own assets if you can, like get equity in companies, buy stocks, own a house if you can. I know that's harder and harder. And yeah, tough, especially if you live in a big metro area. But, you know, maybe there's other stuff you can do. And like, you know, buy MagSeg7 stocks, something, I'm not sure. But I think also, this is maybe counterintuitive, but I would say like, get off the laptop, like go talk to people. Go to parties, socialize, be more human. Because honestly, like, this is the thing that nobody wants to listen to if it's AI generated. Like, it has no meaning, right? Like, no meaning at all. Like, I don't want to read a big essay by AI. I don't want to, like, listen to a podcast by AI. I don't want to watch a movie with AI people in it, you know.

Still, humans are the real consumers. We are the people and living, you know, creature who's consuming stuff, right? And we have the needs and we're the one who's living. So I feel like we're ignored.

It might not always be the case, right? Especially when we have robots or whatever. But I would say at least right now, humans are the only source of meaning in the galaxy. Like, we only care about stuff if other humans care about it. Exactly. Yeah. Or if it involves other humans. Like, I always think of, you know, I'm from England and just moved, and in London, there's piano in the train station in St. Pancras train station. And, you know, I always think about this, like, people will stop and they'll play piano, and it's like always people who don't think, like, wow, you know, that guy plays piano. And, uh, and it's like beautiful, you know. And like, whenever someone sits down to play, even if they're not that good, like, people will stop and watch and clap. And I always just think, like, you know, that's like a tough crowd because people are busy running through the train station. And I just think like, you could literally just get a piano that plays itself and put it there. That is a solved problem, right? Yeah, it's been a solved problem even before AI. You could have a piano play any of the best songs in the world in history, right? Like Bach or Mozart or whatever. And nobody would stop for that.

Nope.

So that's what gives me hope, you know. I think even in a fast takeoff scenario where AI becomes all powerful and it's like the dominant species on the planet now. I still think if we found other intelligent life in the universe, we would want to understand them. We would find it interesting. We'd want to trade with them, even if they were far below our capabilities.

Right.

And so I think humans are a very interesting source of entropy in the universe, and most of the universe has no entropy or nothing interesting going on. Nothing with any meaning. So, at least as far as we know. So I would still think that, like, in a case where you have superintelligence, like, they would still, like, hopefully want to be friendly with us. Because, like, you know, we're not going to compete with them for resources. Like, they could take the rest of the galaxy, you know.

How do you think people should prepare career-wise?

I would say I very purposefully did 3 things in 2020 that made a massive difference. One was I made myself not do any tasks manually. I only allowed myself to do things with GPT-3, which was very painful. And it's much easier to do that today. But it was very important because at first, doing it with AI is unnatural. At first, it's harder actually than just doing it yourself, especially if you're already experienced and already good. Ironically, like the senior people are like the worst adopters of AI because they're already pretty good. They already have great taste. So, so like they, like they look at AI output and they go, that's not good enough. Yet, uh, but, but by the time it is good enough, it might be too late, right? So, you know, I would say that like the junior people that just dive in and they're like, well, it's already better than me. So why, why wouldn't I use it?

We don't know better.

Yeah, exactly. Yeah. So I would say like force yourself to use it. Second thing is to like run actual tests. So I used to like, maybe this is unethical, but like I used to send people things that I had written and then send other people things that AI had written. And, uh, and I would do like a Turing test to see if they could tell it was AI or not. And, uh, that would really convince me, uh, quickly that AI was going to be a really big thing because, um, it, I thought the AI written ones were bad, but I think it was just like self-preservation because I was like, I wanted them to be bad, you know, and it was only when I did the Turing test that it convinced me otherwise. Um, and then, and then I think the, the third thing is just to have fun with it, like, and make space for fun. I, I purposefully only consulted like 2 to 3 days a week. And I was in a lucky position to be able to do that, right? But like, even if you have to work full-time right now, evenings and weekends, your domain, right, that you can mess around with things, you can vibe code stuff, maybe do riskier things that you otherwise wouldn't be able to do in your job. And I think anyone could do that, even with kids. Like, I have kids, right? And yeah, it's really important that you actually have fun with it, because if you're not having fun and it feels like a chore, you're just not going to like learn it as much as you, as you want to, and as fast as you want to.

I know that you have a new book about context engineering coming out. Anything you want to share about it before the launch?

So this one is a lot more technical actually than my first book. My first book with my co-author James was, I would say about like 60% of it was technical and then 40% of it was just like actually quite useful even for people who don't code because it was like in plain English, like prompts are in plain English. It's kind of helpful., there was a lot of image generation stuff in there. This one is like, like a pure coding book. It's like all NDSpy. It's more comfortable for someone who's like maybe with a machine learning background, but I tried to make it as accessible as possible, or more accessible than the documentation, at least. So, uh, yeah, there's a lot of hard-won truths in there, you know, uh, experience that I've had in getting these tools to work. And, uh, I think it should actually be fairly timeless in that, the core DSPy library hasn't really changed in the past few years. It's only kind of, this actually still works. Like some of the old DSPy code I wrote a couple of years ago still works. It just works much better straight away because the models are better. So it is, uh, bitter lesson pilled, as they would say in Silicon Valley. Like it is actually like, it's something where you can learn it and then it's very scalable. Like it will continue to work in the future. I think so that, that, that is like a big concern for me writing books in the age of AI, like physical books that are printed and can't easily be changed. So like, that was the reason I took on this project.

For all TwoSet AI audience, I think we will have 2 free books to give out for free, first come, first served. DM me on LinkedIn if you're interested. Yeah.

And it's already out for preview, I think the first couple of chapters on the O'Reilly platform.

Thank you so much for the conversation. You don't have to be like trained to be a technical person to be one of the most influential voices in AI. You went from running a marketing agency to writing the O'Reilly books that engineers actually read. So, that's a powerful message for our audience.

Yeah, I do have to pinch myself sometimes. Yeah, yeah, yeah. But the imposter syndrome is slowly going, you know, like slowly leaving, but still there, but yeah.

How do you overcome it?

You just keep proving to yourself that you're good enough, you know.

I feel like we compare ourselves with somebody who is better than us in a specific There's always people better than me.

Yeah. Yeah. The problems never go. They just upgrade, right? Like now, like the people I'm comparing myself to are like even more impressive than the people I used to compare myself to, you know? So I think that doesn't change, you know?

Yeah.

Like maybe, maybe it's like Elon Musk or like, or like Jeff Bezos or something like, maybe they don't have anyone to compare themselves to, but, but yeah, I think for the rest of us, we're, we're, we're always going to be doing that.

Okay. Thank you so much. Thanks.