Is Opus 5.5 Better At Writing?

Opus 5.5 communicates more naturally, addressing some of the most common feedback we've heard, like basically slop. Like, okay, it puts the most important information up front and follows the writing rules you give it, makes long sessions easier to follow. Okay, so let's see, let's see this like image.

So it's like 2 parts of that, right? There's like, does it naturally write well and then communicate naturally? And then the second part, it follows Instructions so you can have it write in your own style, I guess, would be the 2 things they're claiming here, right?

Yes. Maybe you need like a prompt to reduce slop, saying also, okay, like maybe it'll just write less slop. Okay, so what I did, like last time we did a writing thing, I vibe coded this like app that does side-by-side review. I'm sure there's like different ways to do this, but this is like an easy way. So I'm comparing like 5.5 to 5, and what I basically— what I'm doing is saying like, okay, I'm going to ask this question. Can you describe— write a short article like an FAQ that tells me, or that describes what AI evals are? You have Opus, Opus 5 versus 5.5. Okay, let's see what happens. It's exploring the data. It's exploring like the context, you know, it's basically having it write an article based on some context, which is kind of Like what you do if you're trying to write something. Okay, you got Opus that started—

It looked like a lot few— Opus 5.5 did a lot fewer tool calls.

Yes.

I don't know if that's good or bad, but I wonder if it read the same amount of context. That's another thing, I guess.

Let's let it generate a little bit and then we can do a side by side. We can see which one we think is better. It took some liberties here. What is eval in one sentence? It's hilarious that it says, what is eval in one sentence, and then give us entire Book on it.

Well, the sentence— I mean, if you use an em dash instead of a period, the sentence isn't over, right? I mean, so starting it first, I do— Opus 5.5 added that extra a short FAQ. I would probably delete the word short. I'm like, because that doesn't seem— people know what an FAQ is. So yes.

Based on the course reader, blah blah. Okay, what is AI eval? So this is where the meat is. An eval is a way to measure your AI application based the way you need it to. It usually isn't one test or one score, it's a process. Okay, so I'm just gonna record the vote and like, let's try something else. Like, let's, let's do something completely different. Like, okay, let's give it a writing task. Maybe let's give it a writing task based on your blog posts. Okay, let's go here. Writing 6 Things I Tried with Jeb. Okay, cool.

Sure.

Let's try that. Write a summary of this post. Make sure it's okay.

Yeah, for people who are skeptical about Jev because classifiers have existed before.

Okay, I'm gonna say be— have a warm professional tone. And I'll even like go further, I'll say like use this plain writing skill. So I'll say like use this plain writing skill from our friend Shreya. I'll just say like, okay, I'll just tell it to look at this markdown file As guidance. So look at this. Wait, let me tell Codex to like— let's make it blind. Make the model names blind and then allow us to toggle them when we want to reveal. Okay, let's jump back here. Okay, Opus 5 is trying to do stuff. 5.5 is still going. I feel like it got stuck or something. 6 things Isaac tried with JEV summarized for skeptics. A little bit too literal of like the title.

Yeah.

Uh, okay. Isaac Flat tested typesafe.jev, or typesafe.jev, a judgment model on 6 jobs from his own work, and Jev scored about the same as Gemini 3.5 Flash on accuracy, answering 4 to 14 times faster. If your objection is that classifiers have been around for years, the post doesn't argue against you. Oh my God. Okay. Like, that's really bad writing. Like, whose objection are we talking about? Like, who's like, you know, it's like, it's taking my— it's like an oversteering based on what I told it. And it's like assuming that like I'm writing about someone that has an objection.

Yeah. It's doing that, which could be— we set it up as fine, but it's like, it comes off as like, I'm immediately responding to somebody who's like super defensive, you know, like they're really defensive and this is their— yeah.

Yeah, this is the kind of stuff that like drives me crazy. Okay. The accuracy— the post doesn't argue against you. It's like, okay, this feels like AI writing. The accuracy numbers are a tie or a small loss in most of the tests. So the case for JEVs rests on response time and not having to build a classifier first. Wow, that sentence is so insanely complex. I have to like concentrate really hard to like get that, get there. It's a lot of, it's a lot of cognitive work for something that doesn't need to be so complicated.

Could be, yeah, it could be super simple.

Okay, summary of 6 things I tried with JEV. Not much better title. If you have built text classifiers before, you'll recognize most of the tasks in Isaac Glass's post, so it's reasonable to ask what JEV adds. Amazing, that's so much better.

Better, yeah.

The post answers with results from Isaac's own work. He used one model, type-safe judgment model JEV. For 6 different decisions about AI and documents. For each decision, he asked Jev a plain question, and the post doesn't mention collecting train data or training a separate model. Okay, it's a little bit of embellishment, but okay. I mean, I would already— we already know it's winning. I mean, I feel like B.

Yeah, I feel like that sentence, like the point that it made there is could be helpful. It would come later. It would come like I'd show the examples and be like, now he didn't talk about how he created the dataset. And so So there's like, it would be like a caveats thing, but it started out strong and then it does this. And then the sentence right after, it kind of jumped into like listing facts.

It gets really bad, like right here. Like all this needs to get deleted. It's going into a rabbit hole really fast.

Yeah. It doesn't need to mention, and this early it mentioned 0.35 to 0.52 seconds. Close accuracy to Gemini Flash might be okay, but like that's getting too specific for the intro when we haven't even Like that, those numbers don't mean anything because we don't even know what the task is.

Yeah, the model trained on label examples to answer one fixed question. Okay, yeah, that's true. It's traditional is the keyword.

The whether an email is spam, I don't think I had that in the post. I think it made up that example. I don't love it because it's kind of a cliché, but not terrible.

Any new example, need a new question answered, usually labeled new examples. Isaac's 6 tasks each take a different question. That's a weird—

that's pretty bad.

That's not good. That's not good. E.g., whether a passage— I mean, how in the hell? I'm like, how can you even like imagine writing a sentence like this?

Like, it's, it's trying to do like a summary, like overview of like, here's the tasks we tried. And I could do that if you were setting it up as like a table of contents to let people do. But then like each of those examples would just be bullets rather than Yeah, it doesn't make any sense.

Actually, it doesn't. Like, did it even say—

Like, tasks don't ask a question. Like, my tasks— that's the issue, is it didn't say. It's confusing because it's like, Isaac's 6 tasks each ask a different— like, the task doesn't ask the question, you know? Like, I asked the question.

We don't even know what these 6 tasks are. Like, where did this— like, when I read this, I'm like, where— what 6 tasks? Like, there's 6 tasks? And this happens with AI a lot. Yeah, yeah, yeah. So like, I don't know, like 6 things, is that 6 tasks? Like, what is going on? And this is the kind of thing like AI like loves to just push information in places where it shouldn't go or it doesn't flow. So, okay, I would say like B or like 5.5 wins, but it's still slop.

Yeah, slop will be a problem for maybe a year or 2, only as long as weak, less capable slop-producing models are still being used. I think we're seeing that 5, 5 is still— it seems better so far with our sample size of 2, but it seems like it's still slop.

Okay, so let's try to produce slop on purpose.

Let's try and make a good Twitter response, like a good Twitter reply, and see if it can do it or if it just comes out slop.

Okay, let's actually read the post so we can like guide it. Okay. RLM is the most important function of my Pi harness. Oh, we can skim it. It's the most important foundation of PyHarvest. That's interesting. It's seeded with late interaction retrieval results. Then you have a setup, you have an iteration loop. Why does this work? Okay, you have a richer shell, keeps main agent context clean, stacks good ideas. Let me riff on this for a second. Okay, so understand the post in detail. Also understand the link from the post of the full blog post and Think about a thoughtful reply that asks him about how does he know that it's better than other approaches, like without RLMs or other coding harnesses? What convinced him that this is the best approach? Please draft a reply. Assume that we are familiar with coding agents in general, and we want to understand sort of his perspective and And then I can come over here, I have a new session. I can copy and paste this and I can say like draft, okay, like a thoughtful reply in the style of a LinkedIn influencer. What was this thing? There was like try alternate mirrors. Okay, I see. It was trying to pull the tweet.

Did we ask it to write a Twitter reply?

I did. I think it's like, it gives, gimme this preamble of like what it did and then it's getting into the reply. Great write-up. A plain sub-agent with RG. This sounds exactly like a bot. Great write-up. A plain sub-agent with RG. Your pilot search also keeps main consciousness clean. Does anyone talk like that? Does that sound human? Like, in what universe does that sound human? I get so upset. Like, when I see shit like that, I'm like, I just, I'm like, let me block this person right now. It's like insane. Like, that is, there's nothing about that, this thing right here, even though we put a lot of effort into making it good. Is really bad.

There's a whole bunch of like sources that I, I look at. I have an LLM judge for like to stay on top of document stuff. It's really funny because I start to learn which places use AI for their posts and I just dread reading them. And so as a result, like I just don't want to read through it because I know there's even a 75% chance it's like kind of slop. So I don't, and then I don't make content about them as much as I otherwise would.

We are just Comparing models. So 5.5 versus 5, we're seeing like, does 5.5 produce less slop? Allegedly it's supposed to produce less slop. Okay, on the right, Opus 5.5. Okay, this is like definitively slop. And I would say this one, great post. Making the traces live notebooks is really nice touch.

Okay, I think it's better than the other. Let's look at the other.

Okay, you think it's better than the other? Okay.

The notebook is live experiment is the part I haven't seen elsewhere. What I can't tell from the post is the counterfactual. 3 ideas stack. Yeah, so I think 5-5 is better. It still is slop. It's still easy to identify slop, but maybe for the next time we come in, we need both failure button.

Okay, this is the one we're trying to write like a LinkedIn influencer. So we're trying to write like slop on purpose. So let's see, you know, according to their Twitter post, it may be 5-5 will write like even more cringe. I mean, it's definitely— okay, like which one do you want to read?

This one is the I'm an influencer prompt that we're about to look at. Okay. Most teams are still arguing about chunk size. And then let that sink in with a point down. I'm reading the left. So that's—

No, but it's like super interesting, right? Because like 5-5 did the same pattern right here. It has a hook and then it has read some shit with an emoji here.

Let's try one more version of Twitter, of social media reply where we take one, give it something. Oh, and 5 has the, it's not X, it's Y.

Okay, let's take one of Brian's. Since Brian is such a good sport, maybe just this one here. This is short. You know something is good when it is simultaneously used to argue conflicting viewpoints. Okay, so you wanna like write something in response to this?

Yeah, so let's do this and let's have the prompt be something like general, like craft a reply that would be interesting to the person who wrote the post and would likely get them to engage. It should be, you know, short and thoughtful. to create meaningful engagement.

So this is pretty short. Now I will say like 5.5 is not shorter. Okay, so let's go back here. Okay, so we got 5 on the left, 5.5 on the right. Okay, so which one?

Start with the left, I guess. Okay, horoscopes clear that bar too. The tell is whether both sides are quoting the same line or just the vibe. I feel like— but we got to pick the better one. I guess I get the horoscope point. The horoscope kind of can mean anything. You can interpret it any way. The second sentence is super bad. Okay, or it's a Test my tiebreaker. Both sides painting.

I don't know, man. It's like—

This is, yeah, this is really bad. So I think we do not have to worry about engagement bot. That is not detectable.

Yeah, I don't think we have to worry. Like 5.5 has killed slop. They both fail to me. I know which one is better, honestly. So I'm going to say they both fail.

Title and abstract for conference proposal. That's a good one. That's really good.

Yeah. Okay. Yeah, I'm just pointing it at my evals material. This is kind of trying to make it realistic, right? Like when you write something, you're like, you're trying to point it at some context you have. And I'm saying like, okay, write a title and abstract for a conference proposal, conference proposal for a talk. It's kind of a weird sentence, but okay. Are we ready to go?

I mean, that is as good context. I mean, you got a whole book, so that's as good context as you could hope to give for something like this.

They're both running on high effort. default Claude code prompt. And then we'll see. We'll see, like, if one— your LLM judge is a classifier, start measuring like one. It's not bad, although I wouldn't try it. Like, it's too, it's too hypey to me.

I feel like it could say measure your LLM judge like a classifier, and that would be better.

Now their post says, says that 5.5 is always shorter. I don't know if I feel—

Okay, so at the bottom The left one had an attendees leave with, which is on 5.5.

This one has attendees leave with too, but it's a lot shorter.

A checklist for building a judge and a way to see the judge's AI run versus a criterion for choosing code checks over—

No, I already don't like this. It's too much. It's too much. Like, I don't like— URL image judge is a classifier. Start measuring like one. Okay. Gave me some alternates. Interesting.

That title actually somehow is better with 5 to 5.5. 5.5 is your LLM judge is a model, validate it like one. I mean, they're both kind of bad, but at least the left tells you.

Yeah, let's do the abstract. Okay, so 5. Most teams write an LLM judge prompt, eyeball a few outputs, and start reporting the number. That number is unfalsifiable. Okay, I would never write like this. This is very confusing. The judge itself is a classifier with 2 independent error rates, and nobody has measured either one.

You haven't told us what either one is either.

Yeah, it's just so— I'm like, when I read this, there's so many problems with this. Like, I don't even— this is so bad. This is like classic slop. There's nothing good about this.

This talk walks through the full pipeline for building a judge you can defend using customer support agent as the running example.

Let's read Opus 5.5. More and more teams measure quality with the LLM judge. A prompt asks a model whether a response is good, and the answer goes on a dashboard. Okay, it's trying to like be too dramatic. It's trying to— it's not— it's a little bit better than the other one for sure, but it still sucks. Too often nobody checks that judge against human labels. This talk shows how to build a judge whose numbers you can defend. The running example is a customer support agent that tells shopper your refund is complete. Wow. Okay. It's very parallel in its structure. Like, it's like really interesting that like wants to say the same thing and organize it the same way.

So yeah, I wonder what they did. Did they change the data that it was trained on? Because it doesn't seem like significantly because it's still mirroring the same thing. Was this just training? What did they do to 5.5? Like, what did they do differently to try and get it to write different? Because it's still so similar. It doesn't seem like they made some improvements, it seems like, but it doesn't seem like they made like a complete, like, major change. Because of how similar they are.

They're extremely similar. Like, the title is similar, the first sentence is fairly similar, and then the structure is like, we start before a prompt, you start before a prompt. Okay, we looked at how many examples? Let's see, like, we looked at— so we have like 3 wins.

We had 3 wins, 1 where both failed.

I'm gonna say I don't like either. I mean, they're both like, just, you have to edit the hell out of both of them.

There is enough slop in 5-5. Still that we would need to do a lot of editing.

Yeah, I mean, like, we have 3, but then we did 6. I think like we passed 3. So it's like very inconclusive. I don't think we can say like 5 is— 5 is like definitively better. And maybe it was like a little bit better in some of them, but I don't think—

Can we go back and review the first one we did and see if— do we hate it more now that we've seen a few more examples? Or do we still feel like— like, will we fail them both if we read it again?

Yeah, I mean, dude, A is definitely— left-hand side definitely lower. It's clear on this one.

The basic unit is the evaluator. The reader defines it as a procedure that maps a trace in relevant context to a judgment about a specific property. A trace is a full record of one request.

It starts okay, but then like, yeah, as you go down, it starts to get bad, right? Like, I'm sure like if I kept reading this, I would start hating it too eventually. I'm saying like, okay, just from the intro, I feel much better with FiveFive in this particular example.

I guess I feel like in all of them, I thought FiveFive was better than the other, but I didn't, I didn't think any of them were Great.

Okay. You thought even for this one, this conference talk one?

Yeah. I mean, I liked it better. I guess when I look at this, my thought would probably, like in both cases, I would probably start over. So like we start before writing any prompt.

Slop is not dead. Uh, 5.5 might be a little bit better in some cases, but it's not like a landslide by any means. I think slop is still here. Like it's very much here. Like it has not been solved.

For the ones they reviewed, the conference talk this person reviewed, the titles were mdash-written and the abstracts were mostly generic. Like they'd asked an LLM for ideas on topic X and all gotten the same responses back. I ended up prioritizing abstracts that had specific enough examples to make me think they'd written it themselves.

Fascinating. Wow. So like all the abstracts were the same.

Yeah. I mean, we reviewed 6 and it already feels like Painful.

All right, thanks. All right, see ya. Real quick, if you're building an AI product and you're stuck on how to improve, we put together this free resource on evals to help you. It draws from more than 1,000 hours of working with companies on building AI products, and it's customized to what stage you are at. So this is how you can use the guide. Go to bit.ly/evals-guide. When you open the guide, it'll look like this, and what you do is you select the stage that you're at. Maybe you're new to evals, you don't even know what evals are. Maybe you do have evals, but you don't really know what to test. Or maybe you don't trust your eval scores. Or maybe you think your product is just too hard to eval, or it's too time-consuming or expensive. Whatever the case may be, we have a detailed guide customized for what stage you're at. We've worked on it for over a year and are giving it away for free. Go to bit.ly/evals-guide and we'll send it to you.