Why The Prompt Matters Less Than The Context

We have this prompt guidance. Claude's Fables 5.1 writing is generally a step up from earlier Claude models, cuz like writing is the most painful part about using AI. I think we can all agree. With fewer stock phrases and less unexplained jargon. In some cases though, its prose is denser than Claude's Fables 5. Sentences run longer, blah, blah, blah. An instruction that defines the intent, pattern, manner, prose helps. And it says like, use this, right? And so if you look at the prompt, if you read the prompt, it says like, Mannered prose substitutes metaphor and flourish for a direct statement. Instead of a parameter worth varying, the mannered writer produces a dial worth turning.

And honestly, the funny thing is, is that like both of those, a parameter worth varying or a dial worth turning, both of those are bad in my opinion.

The phrases exist to display the writer, not to convey the idea, and readers can tell. That sentence seems okay, I guess. That is why mannered prose irritates. Oh my God, it makes the reader work harder so the writer can perform. I mean, that is pure sloth, let's be honest. And then it is also imprecise. Metaphors drag in connotations the writer did not choose and cannot control. The fix is— here we go again— to say what you mean. When a literal phrase is available, use it. And they said like this short version also tends to work, which is Fascinating. But I'm like, when I saw this, I'm like, wow, okay, why would dropping slop into a system prompt work? Like, it defies my intuition. Like, kind of like in-context examples. Like, if you have an in-context example that's slop, you would think that that's going to sort of steer the model towards more slop. If you have slop, even if the slop is saying stop writing slop, you would think it's like not The best thing. So I think this is what we want, we wanna find out right now.

Yeah, I would think that whatever's in its context, it's, it'll create more of that. I hope this doesn't work because I don't wanna always be writing my prompts like that. Like, I don't, I don't want like the future of prompt engineering for us all to like write this to get our agents to do things. That'd be terrible. So we should, uh, try it.

So we're comparing no system prompt, unmannered, and plain writing. So these are 3 different ones. All right, let's have a bake-off.

Okay, so, so far unmannered is, is the loser for me.

But the, the bones are better here. Yeah, I can, I agree. I would say let's call a C a winner.

I agree. Yeah, I agree.

I think I like B a little bit more. If you want to put a gun to my head, I would say, okay, let's go with B.

It is very interesting that the unmannered prose is so close to like a good writing prompt or good writing prompt. What we thought was good. I think the unmannered one did a little bit better because it said 17 over 22 versus 14 over 22. That made it a little bit easier to know. And it also included one last fact. About the cost, which is nice. And then like the finding, grep versus semantic search, same accuracy, cheapest run, is a lot better, I think, than grep versus semantic search over the same OCR output, same accuracy, cheapest run. The bottleneck was never search. I mean, it's true, but it's like too wordy. It's not good.

Yeah, the slop is here.

I don't know.

I think B is the best.

I think so too.

I think what I'm learning so far is like there is no winner all the time. Necessarily. Like, you could have good things from each.

I think it's, it's, it's, it's very interesting and surprising to me that there's not a clear winner.

If I was to try to do this for a serious task, I might, I might say like C. Uh, I would say like C is the better bones to it. Yeah, because it's not great.

It's like, yeah, if I think about how I'm gonna take A and then like edit it to be good, it's like, I think, yeah.

Okay, so I don't know, man. The, uh, unmannered It kind of is way stronger. I mean, we only looked at 5 examples, but I thought it would be like not winning any of them.

I thought it would— I thought it was just gonna be a joke. I thought it was gonna be terrible. Is it just that the prompt for writing doesn't really matter? Like, could we write a prompt that's like, write like a good writer, and is that just as good as all the other prompts?

No, this is inconclusive. This is not enough data to conclude anything. It just tells us that it's all over the place.

And I, I learned that it's a lot closer than I thought. I thought for sure that the unmannered would just never be better.

Yeah, I thought there's no way that's gonna— I thought like, oh, like my own handcrafted prompt that's like has a lot of rules in it and it's like more carefully considered is going to be a lot better than, than this slop. I'm not really sure that's the case, and I'm not sure like writing a prompt at all is like— in some cases, writing any prompt It's like helpful. And the fact that you have this piece of slop and then they said this short version also tends to work, which is just the word manner of prose. That's all it has. That does like kind of fit in with your hypothesis. Like maybe the prompt doesn't matter so much, which is really interesting. Yeah, it's like really the context. I think that was steering it the most.

I was actually kind of skeptical about doing this because I was like, we're just going to come on here and it's going to be like a joke stream and talking about how this doesn't work. And it's kind of obvious what's the value, but it surprised me.