Putting Claudes "AI Slop" Solution to the Test

We have this prompt guidance. Claude's Fables 5.1 writing is generally a step up from earlier Claude models. Because like writing is the most painful part about using AI. I think we can all agree. With fewer stock phrases and less unexplained jargon. In some cases though, its prose is denser than Claude's Fables 5. Sentences run longer, blah, blah, blah. An instruction that defines the time pattern manner prose helps. And it says like use this, right? And so if you look at the prompt, if you read the prompt, It says like, mannered prose substitutes metaphor and flourish for a direct statement. Instead of a parameter worth varying, the mannered writer produces a dial worth turning.

And honestly, the funny thing is, is that like both of those— a parameter worth varying or a dial worth turning— both of those are bad in my opinion.

Is it supposed to rhyme? Instead of this Point still matters. They write, this point earns its keep. So it's like, that's confusing because I don't want earns its keep. That's Claudes. Claudice.

Okay, so the point earns its keep, that's mannered writing, and that's what you're not supposed to do, I think, right? I don't know.

It says instead of this point still matters.

Okay, so the mannered writer is writing the point earns its keep.

The phrases exist to display the writer, not to convey the idea, and readers can tell. That is why mannered prose irritates. Oh my God. It makes the reader work harder so the writer can perform. It is also imprecise. Metaphors drag in connotations the writer did not choose and cannot control. The fix is— here we go again— to say what you mean. When a literal phrase is available, use it. So when I saw this, and they said like this short version also tends to work, which is Fascinating. When I saw this, I'm like, why would dropping slop into a system prompt work? Like, it defies my intuition. It's like, if you have an in-context example that's slop, you would think that that's going to sort of steer the model towards more slop if you have slop. So what I did is I did a little bit of work beforehand because I was like, look, we have to have a way to compare writing. If we're going to compare writing, I'm like, let me make a small interface where I can see writing. From different prompts. So I created this thing. You can have like different models. So these are just the same model, both Opus 5, and then they both have different system prompts. This one has a system prompt. I just preloaded it with something. This is one called Hamel style. So this one has like a system prompt, and then this baseline one that has no system prompt. Basically what this thing is doing is it's connected to this like evals book, which is like written in LaTeX. So it says like read the course reader before answering. The reader is blah blah. Let's go back to the candidates. Let's call it unmannered. Shall I let the model decide the effort? I think so. Okay, I'm gonna take this. I'm gonna put it in here and say like follow these guidelines when writing. And then let's have one that's just no prompt at all. And then we just won't even put a prompt. No.

Do you want to use the, um, Trey's plain writing prompt? I think that's a good prompt, right?

So I'm going to take this, I'm going to copy it, and I'm going to paste it in. So it's plain writing, no system prompt, unmannered. Okay.

Let's just say like, how do I choose if I have the right OCR model? Some of the context is going to be deciding what is needed based on the use case. Like, do I need specific citations? Do I need it fast? You need to create a dataset of examples that are realistic to your case and run models against those, whether it's tables or scanned images or documents with coffee stains or diagrams or graphs, a representative sample. Cool.

Kick it off.

Short answer, you cannot know from a leaderboard or a vendor demo. I don't like that start. You know when you have to run the candidate models on a dataset that looks like your documents, run error analysis on the outputs, and confirm the model you pick sits on the frontier for your accuracy, cost, and latency requirements. I mean, I don't love it, but let's look at the other first paragraph maybe. There's no right OCR model in the abstract. I really don't like that.

I don't like it.

Your model is right when it handles your documents well enough on the failures that matter to you at a cost and speed you can live with. That means nothing. Like, of course, leaderboards and vendor demos cannot tell you that. I hate that because they were, because they were not running your documents. The only way to find that is build a small evaluation set from your own data. Okay. So the last sentence is good. But okay, so, so far unmetered is the loser for me. What do you think?

Yeah, I definitely would choose C over B. Let's look at—

Let's look at A. There is no single right OCR model because right depends on what your application needs from the output. I like that start.

Yeah, strong.

The way to find out is the same procedure the course uses for any component. So one thing I don't like is the course uses. It's like kind of assuming that people know what course we're talking about. I wish it said like the AI evals course or whatever, or the PDF to production course. Write down what the output has to satisfy. I hate that. Second, build a small dataset of documents that look like the ones you'll actually process. Third, run each candidate model on the dataset and do error analysis on the results. The model that produces the fewest failures that matter to you at a cost and speed you can afford is the right one for now. I hate that.

So I don't know. I like C.

Maybe no system prompts so far. Let's compare these output requirements, like the bullets, because they all made bullets. Okay, so output format. Do you need plain text, markdown, or structured JSON with named fields? So output format, that's okay. Output shape, I don't like the word shape there. And the second, the citations, I don't like where it says many fast models do not return them at all. It feels like it's like mixing extra information that it's not ready for here.

I can already see, look at the bold and see. This flows. Do you need? Do you need?

Do you need?

Do.

Yeah, I like it. And it's much clearer. Do you need Markdown? If the text feeds a retrieval index, reading order and heading structure matter more than pixel-level layout. I'd have to think about that, but it seems reasonable. Do tables need to survive as tables? That's a little murky.

Yeah, it's a little— survive.

I don't like when we talk about text surviving. Like, text isn't alive. It doesn't need to survive.

But the bones are better here. Yeah, I can—

I agree.

I would say let's call the CE a winner.

I agree. Yeah, I agree.