Opus 5.5 Made Claude Code Usable Again

Opus 5.5 is out. Anthropic says it is about Fable 5.1 levels on most work and about 40% cheaper to run than Opus 5. Are we ready to be hurt again? That's a question that we're asking today as we go over some of the Opus 5.5 updates coming from the Anthropic team. We'll go over some benchmarks. This is your weekly AI Clarity. I'm your co-founder and co-host, Robert Ta, CEO of Clarity by Epistemic Me, joined by my co-founder and co-host, Jonathan McCoy, 12-time founder and CTO of Clarity by Epistemic Me. We build digital twins and we also every week make it to your inbox and wherever you like listening to podcasts to give you signals from the front lines of building our own AI-native startup to help you AI better and build better AI. In our podcast, we go over the latest AI news. We go over some memes, some of them you send to us, we screen them, we show them on the pod, and we go over what's the latest meta. So that's our What's the Meta section where we focus on the meta for AI builders, being AI builders ourselves, building on the front lines of this AI era. Today we have an interesting setup. So Jonathan and I are going to test a new little autonomous closed agent loop that we want to stand up. So we have an agent that helps me write the podcast script. It is called our Podcast Scripter agent. Or scriptwriter agent. And we just thought, hey, it would be kind of cool because me and Jonathan have digital twins. Our whole Clarity API is self-models, particular opinionated take, drawing in the principles of neuroscience and philosophy, the epistemics, uh, to model the self and everything that makes the self. So in our case, we have digital twins of ourselves, me, Robert, and Jonathan. Uh, so we actually asked our twins for takes on the AI news and what's the meta. For this podcast, let us know. Uh, we're gonna read some out and then we're gonna see, was this good or was this bad? Sort of a dogfooding Build in Public segment that we think would make this a little spicy and fun. Let us know what you think. This will be our first time doing it. If y'all like it, we'll keep it going. All right, so Opus 5.5 is out and Anthropic says it's at Fable 5.1 levels on most work, 40% cheaper to run than Opus 5. Opus 5 was kind of Terrible. And this tweet came out from Nat McGalessy.

McGalessy, I think.

Opus 5.5 is way better than Opus 5. Sorry about that model. Please try this one. 1 million views, a bunch of retweets, 11,000 likes. Uh, this person here, extremely impressive after the upset of Opus 5. Might have to try running it again daily. Gotta admit, you guys cooked. We'll test out a plan for you, Nat. That's cruel towards Opus 5. Models are not disposable. At least Opus 5 spoke up. I can't tell if this person's serious. That is a researcher from Anthropic AI, was previously at OpenAI and DeepMind. So, you know, a heavy hitter over there apologizing for the atrocity that is Opus 5. We've shat on Opus 5 enough on this podcast. We don't need to shit on it more. Look at our old content for that. In the past month, we have extensively tried it and found it wanting. And Jonathan, I guess before I turn it to you, Before launch, users really complained about Opus 5. We complained too that it was verbose. It was very low agency. It would forget it could do things. I have to remind it like, hey dude, you could do this. You have an API key right there.

You could do this thing. You've done it before.

It had slow first replies at like max effort. And then the review-fix loops and goal loops I was getting out of it wasn't as performant in real-world cases working on full-stack backend and frontend code as compared to other models, including those from ChatGPT. So, What we're going to do is give our takes, and then we're going to go to our twins and then share their takes. And I'll read those out and what informed our digital twin takes on these. So Jonathan, what's your take on Opus 5.5 and how did Anthropic handle this release as compared to the Opus 5 release?

Yeah, I guess I just reflect what everybody else is kind of saying and feeling. I'd stopped using Claude for a lot of stuff. Just my Fable tokens were the only ones I was really using. And I feel like it's back. It's back in a big way. And remember, Opus 5.5 was released on the same day as GPT-6 Sol. They're both very good models. It's like one of the first times I can remember both companies released a great model update on the same day. And I think that the Opus 5.5 is just more fun, more impressive, more capable even than the GPT-6 Sol, which is also a good model. It made me so much happier, man. I don't know about you, but I was just like, I can finally just use, and I can finally just use Cloud Code again and get work done and not like, you know, and it felt like the magic of Opus 4.6 back but better. And I think that was like, you know, almost a universal experience amongst developers that are using it. So good.

So Cloud Code, Cloud Code's made it back to your daily driver for some activities or?

Yeah, I can rely on it for more tasks. Um, you know, one habit I've gotten into is that if I have a Codex session, I can just say like, hey, Opus 5.5, take that Codex session and then continue it and finish it off or something like that.

So.

use the existing Codex session as context. I had almost all my sessions over at Codex, and then I just could easily move them back, you know, just by having Opus 5.5 read the files on my hard drive and like continue them. And I was really impressed with what it got done. So, and I think like I just want to play with it more and do more stuff. And I think both getting work done and then some of the new stuff you see, like people making videos and its ability to use new visuals, is something that I want to play with more. And I think the feeling of the relationship with Anthropic that we all have, right, is like, okay, who are these guys and why did they give us such a shitty model? Do they really know what they're doing? Can they be consistent? This is a big trust bridge that they just threw out. Uh, yeah. And I know I'm not the only one feeling that. So this is cool.

I shat on Opus 5 a lot. I think deservedly so. I think they needed the feedback to be like, guys, this is a developer tool. Like we developers, we, us builders, we need to build production-level applications with it. It was really great to hear. The Jev founder of recent virality fame with his product that is clear product market fit. They're like 440 times cheaper, 190 times faster for certain use cases with their System 1 models that's come out. Watch our previous pod on that. He said something like, we're building for prod, not God, or we're building prod, not God. And Jonathan and I were like, thank God. Like, that's— we are, we'll build, we're building production stuff like artificial superintelligence. That's cool to like speculate on, but we need to contribute to the economy. One little detour, I guess, I'll make is, let me share a couple videos that I have created with very minimal prompting that I'll show you from Claude here. So I just went to Claude and this is Claude Code. Let's go ahead and just do Claude dangerously skip permissions. And I said something like, create a flashy demo as a modern startup for our modern startup showcasing Verity API and digital twins. Powering company brains for teams centered around the tagline, right context. So you might say something like this, create a flashy video demo for our modern startup showcasing Clarity API and digital twins powering company brains for teams centered around the tagline, right context, right time. And then you might get something like this. So that's one example I liked. There was a couple other examples to show you. So I had it go and actually create a bunch of like old world, new world level. Here, here's another one. So it kind of just went off and created this. It ascertained our design system and brand guidelines from the repo from where I started Cloud Code, but it went and actually made this whole thing. I didn't tell it to do anything. It used our logo. You saw the prompt that I used. It was pretty simple. Once I liked a few of these, I prompted in, oh, I like this video, I like this video, here's why. and hey, use this tagline. And it's actually coming up with pretty good stuff that I could imagine putting some paid ads behind and getting a level of conversion. Like, this is pretty damn good for one-shot kind of spray and pray. What are your thoughts, Jonathan, of this use case of Opus 5.5 to just create like a marketing video?

It's really impressive. I would say the visuals and audio like surpass expectations. And then like the storyline is still like, Feels like a little bit of slop, you know, but I think that could be reined in, you know, with more edits and stuff like that. But hey, I mean, this is impressive. I haven't seen a model do this yet. First model.

This is kind of wild. So for any product marketers out there or product managers that got to deal with creating video assets for your new releases or attracting users, sign up, waitlist signups, whatever, check out Opus 5.5. I'm just showing a quick Hey, like all I prompted was something like create a flashy video demo for our modern startup showcasing Clarity API and digital twins powering company brains for teams centered around the tagline, right context, right time. And you see that it came up with this whole video, took our design system, our brand guidelines, all the typography, even our asset here, our logo for Clarity and animated it. The storyline, as Jonathan said, it could use work, but this is sort of Great for, you know, 30 seconds shooting off this prompt into the void. And then it came back with like a really junior product marketer's output. And I think after spending a little bit of time on this, and I actually quite love this clarity, your company brain, right context, right time. It did get this intent right from me and used a little bit of our like hidden background from the website too. So that's really cool. But I could see like this easily getting better, especially on an evals loop where I test these videos out. to get real signal, but the capabilities more so to demonstrate are definitely getting there through time. This is not something I would've expected out of Opus 5, that's for sure. So Opus 5.5, let's go to the Anthropic release page about this. So I, first of all, I thought this was really cool. When you refresh the Opus 5.5 page, you get like this cool animation. So always a little bit of a product nerd, but working backwards from what does the customer see? Um, I thought this is really cool. I also like nature, so I like their, their focus on nature. We're introducing Opus 5.5, the first model in our new Claude 5.5 family. It performs at the level of Claude 5.5 on most work and costs 40% less to run than Opus 5, which was garbage. I think that's a sub-tagline.

I can't see, but it's there.

Claude Opus 5.5 is our first release since we called for pacing the frontier. We're going to get into that in a little bit. It was tested Before release by external evaluators, including Frontier Design and METR. On our automated behavioral audit, the most comprehensive alignment test we ran, Opus 5.5 is the strongest performing model we've tested to date. It also comes with the safeguards we've developed for our most capable models. So a few improvements you can expect from Opus 5.5. Number one, performance. Opus 5.5 is a major step up from Opus 5. It's the new leading model and early testers saw large jumps. One tester completed a 680,000-line code migration in less than a day, work that would've taken an engineering team weeks in the past. It's good at finding and fixing inefficiencies in software. When we asked to cut load times across every page of a web app, Opus 5.5 succeeded 39 of 40 times, while Opus 5 made smaller improvements that also altered the app's behavior. Speaking to my pain points of using Opus 5, yes, definitely went off the rails a bunch. Number 2, Opus 5.5 is good at safety. So 5.5 achieves the best scores of any model to date on our automated behavioral audit, our alignment suite that tests Claude across thousands of simulated scenarios. It is much less likely than recent models to take hard-to-reverse actions or act outside the boundaries it's been given, and it's more resistant than Opus 5 to prompt injection. Full details of the evaluation are available in the system card here, which is over here on the website. You can get to the system card. We'll throw this in our show notes. The Opus 5.5 system card. Going back to the release, because Opus 5.5 is comparable to Cloud Mythos 5.1 in biology and cybersecurity, we're deploying it with safeguards similar to those on Cloud Fable 5.1. Vetted orgs can apply today to our Life Sciences Verification Program to use 5.5 for biology research. In the coming weeks, we will also be expanding access to our Cyber Verification Program and verified cybersecurity practitioners will be able to use 5.5 for their work. Number 3, it's better at cost and at speed. So Opus 5.5 requires less compute to serve than Opus 5, and its pricing reflects that. Our tests showed that at default settings, it will cost 40% less than Opus 5 on typical workloads. Input and output tokens are $4 and $20 per million tokens, 20% less than Opus 5. Cash reads are $0.20 per million tokens, 60% less than Opus 5. 5.5 also generates output more than 30% faster than Opus 5. So really seeing getting to more value-based outcomes, at least directionally here, where they're now trying to compete on utility, cost, speed, instead of like, hey, we just cover more use cases, which was what they were communicating last time with their 5.0 release. Communication. Opus 5.5 communicates more naturally than prior models. I would hope so compared to Opus 5. Opus 5 was really terrible at communicating. Like we had a joke on the podcast that it was like Opus 5 was like beat in prison or something and then let out of its cave. So I will say 5.5, definitely better at writing back to me. Uh, and it seems to have more agency. It's more of a delight to work with. Early testers found its writing clearer and easier to follow, which addresses some of the common feedback we heard about Opus 5. It puts the most important information up front. And its style makes it a better work partner over long sessions. As one early tester put it, it writes the way I do. In our own use, this has made Opus 5.5's work easier to follow and check, which is a safety benefit as well as a practical one. So we can see, see here some of the benchmarks of Opus 5.5 versus Fable 5.1, Opus 5, GPT Astra 6, GPT 5.6 Sol, Agentic Coding Terminal Bench 4.0. It beats the competition pretty handily, nearly 2x-ing SOL, like 1.8x-ing. Agentic coding, 54.4% versus Fable 5.1, which is at 50%. Astra GPT-6 at 53.3%. Agentic coding from CursorBench, 57.8% versus Fable 5.1 at 51.8% and Opus 5 at 46.6%. And a few other things here I'll note, this one. This always comes up. Multidisciplinary reasoning, humanities last exam benchmark, 67.7% with tools versus 56.6% and 63.6% of the past models. Agentic scientific research, which has been another focus of the model labs recently, 58.7%. But Astra beats this one at 64.6%, a whole 6 points or so difference. Computer use has gotten better very marginally. At 81.8% versus 80.7% and 74% of 5.1 and 5 respectively. And we could see some of these accuracy versus cost benchmarks where when we have Opus 5.5 on at max, the score is higher and lower in terms of cost versus some of these other models like Opus and Fable, the prior models. And then showing it's, hey, even at similar price point, The terminal benchmark is at a higher output for the benchmark. So accuracy versus cost, they're showing that the latest Opus 5.5 model is definitely at the frontier. Jonathan, any other thoughts on this Opus 5.5 release?

And some—

if you could describe some of the work you've been doing with it, I think the audience would be excited to hear that.

Yeah, I mean, uh, one, one thing I'll say that, uh, is a writing ability and readability from writing. Is just, is a felt difference in it, right? I mean, I think I also noticed this on the Every team's post. They had banished Claude from the writing room. They have a specific benchmark about like how well the models write, and they do a lot of writing, the company Every does. And now with, with 5.5, it's back allowed in the writing room. In fact, it's the best performing model that they've tested. So I've, I've been enjoying that. And the type of work I'm doing is, is back to like how I used Opus 4.6 and Opus 4.8, just like My go-to model. Like, this is my workhorse. This is, this is the guy that I can count on to like start a project and get stuff done. And it, it's such a better feeling of using the product of Claude Code. And I'm, I'm kind of like wondering, like, um, you know, what did, what did Anthropic do, you know, sort of internally? It sort of looks the same on the benchmarks as the, the progression of Opus 5. Like, you know, um, I'm kind of curious about like the internal conversations they had and how do they sort of like fix this and If it's model releases in the future are going to be kind of bumpy like this and we should expect that, or if Anthropic's actually learning to produce consistently better models. I kind of think it's more the former. It's going to be rock. Some of these models will be really reliable and other ones won't in the future. And we just kind of take them as they come out, but we'll see. So anyhow, I'm happy using it for my day-to-day right now.

Just speaking to your last point, I bet this is how it actually has gone, which is leadership changes direction and then that kind of ripples in the pond of the organization. A lead dev like Boris is now put on like, okay, how do we like solve for this? I don't know when people prompt it for biological weapons and hacking and stuff. And then he gets put on something else. He's a strong driver of the team. And then the team's like, they're not dogfooding as much for the development case or getting shit to prod case. And then like a few sprints creep by and now the model is Opus 5 and it's bullshit. And then maybe, and then maybe the marketing team wasn't super aligned with it. Maybe they didn't get enough test cycles. I don't know. Like, this is just like growing pains, company operational excellence stuff, I would imagine. And they're having to navigate tons of change all the time. I bet that is sort of probably feels like whiplash internally and they gotta externalize products. Right. At the same time and keep up with the cadence of releases. So I bet it's some very human cat hurting level shit like that. I don't think there's some, 'cause if we, if we sort of magic wand the Anthropic's this org that like does things, it's like at the end of the day, it's a bunch of humans like running these agents and it's still humans putting their names on releases and shit.

You know?

So let's go to our, our twins takes of this. So twin take, let's do a twin check. So Robert's twin, what did my twin think of the Opus 5.5 release? Yeah, honestly, I'm glad he said sorry because Opus 5 was rough for us. That's in regards to this tweet here by Nat, who's an AI researcher at Anthropic. That apology is the first bit of trust I've seen come back, but a tweet is not evidence and neither is their own table. And Anthropic basically says so on the same page, which is benchmarks are a weaker guide now. So for AI builders, the lab's words earns a trial run, not a switch. You run your own tasks against your own quality bar and the price drop roughly 40% cheaper just means the trial costs you less. This is, this sounds terrible actually. Like I think this, I would never say the lab's words earn a trial run, not a switch. I would say something like, so for AI builders, try it on your own shit first, then make your own call.

All right. So what informed Robert's twin take? So my twin's take on August 25th, we talked about Opus 5 Trust. They pulled my quote here. Benchmarks are benchmarks. It informs this belief statement, 225 in my self-model. The evaluation we have products should be based on real-world performance rather than just benchmarks. Actually, yeah, I hold that. That's a true belief that I hold. Quantifying subjectivity at minute 6 of that same transcript. This matters because the benchmarks that you see out there like Grok or OpenAI, those don't meet domain-specific cases. That informs belief 359. The benchmarks provided by foundation models Do not meet domain-specific cases as they failed to test a myriad variations of user queries in specified verticals, health or longevity. Yeah, that's pretty cool actually. I processed a bunch of our recent podcast transcripts to bootstrap our soft models for this episode and get them more up to date. Jonathan's twin. Jonathan, do you want to read this? This might be funny.

Yeah. I mean, the apology is kind of refreshing, but it also tells you the lab's word on the last model wasn't worth that much either. I find the last one unpredictable and a lot of other, a lot of my own use cases, like a step back that I hadn't before and no benchmark table warned me about that. So I think the lab score gets your attention, but it's probably worth trying out on your own task this week and that's it. And that's, it's not a reason to switch. Anthropic saying the benchmarks are a weaker guide now is, you know, what we've been saying about those scores that not matching the real usage. So what does your team actually have? that tells you on your own work that it gets better.

What do you think of this take? Yeah, tell me more.

I think the take is pretty much right on. It's probably literally based on things that I said before. The voice is a little off, but yeah, it's not a decent representation of how I would respond to the situation.

Yeah, it's kind of cool. Yeah, because we have talked on the pod plenty about how a lot of our guidance and philosophies around this stuff is You got to take some cycles and just try it out on your own flow of work, your own domain-specific problems. You could see how Jonathan's twin is evidenced from these episodes and the belief statements. So like this one, Opus 5 trust and avows, and they release Opus 5 and then it breaks 80% or regresses 80% of those use cases. Belief 236, which is informed by that, the release of Opus 5 has regressed performance in many use cases compared to Opus 4.8. So that's like previous Judgment. So for my twin, I think it, like, the take was good. So the meaning and decision-making and reasoning and judgment of my past from the recent, like, 10 episodes of the pod, I think really hold up well. What it missed, I think, was the tone. So gotta do some more passes and fine-tune the actual writing style to sound more like me. Same for Jonathan, I think. Did it hold my take more strongly, less strongly, or about the same as I did? Going back, let's see, my Robert's twin. Did Robert's twin do well? I, I think I changed this part. So half of it was like pretty good. Even in terms of like tone, I think this is good except for this. Maybe, I don't know what this table thing meant, but a tweet is not evidence. Yeah. Yeah. I think that like Jonathan said, take was good, reasoning good, but taste-wise not quite there with the voice. And same thing for yours.

Yeah, I was particularly impressed by the evidence it pulled and how relevant it was to the situation. So I think that's Like pretty strong. And then, yeah, the voice is a little off, to be expected, you know.

Next, next pod, it's going to get better from these evals we're doing live, guys. So for the audience, me and Jonathan are in the business of helping teams make their AI products better. So we figured, why not demonstrate this on the podcast where we literally read from our own digital twin tech to improve our own podcast script writer agent runs? This little segment where it pulls from our digital twins for the takes. So we can just demonstrate what do our digital twins even do, and then we get on our own loop making the product, the digital twins better of our own digital twins. So next, next pod, this should be better. Okay, let's go to the next thing. If you're an AI builder and you are looking at Opus 5.5, try these in your first session. Hand over a whole task, define done and when to check in. This is from Claude devs themselves. Number 2, drop Think carefully. It always thinks first, so don't worry about that. You don't got to product along anymore. After a long run, check what it needs to go further. And here's their playbook: getting the most out of Opus 5.5 in Cloud and Cloud Code. How to prompt Opus 5.5 steer a long run and check your results in Cloud Apps and Cloud Code. So as I said, try those three things first. How to ask things? So say what done looks like, then let it run. Why does this matter on Opus? 5.5, it keeps going on long multi-part work better than Opus 5 did. So the best gains are from you just defining done. Here's a way that you could, you could do that. Say what done looks like. The whole task might be migrate the payment endpoints from the old client to the new one. The finish line done means every endpoint uses the new client, the old client is deleted, and test suite passes. When to stop and ask? Only if a test fails or a reason you can't explain. So that might be your whole steering for that case. Adding to a running task. If you remember something mid-run, you could type a follow-up while it works. And runs are longer now and restart may cost more because of it. So just type follow-ups in Claude Code and press enter while it works. Another, another thing that I find useful, and I'll just demo live right now, is for anybody using Claude Code out there, let's say it's like going. If you want to ask it a question, but not interrupt its flow of work, you could use this /btw. Hey, by the way, how many videos have we created? And then it'll actually do a little side thing without interrupting the main agent flow and give you this slash by the way response. So 15 videos numbered 00 to 14. So if you're an AI builder and you're using Claude Code and you typed in this, you yeeted some long prompt or command into the void, and then it's spending tokens, you don't want to interrupt it. Maybe it's goal loop or it's cycle that you have it on to improve whatever or build whatever. And, but you do want to ask that rich context something you could do /bt-dubs. Give that a shot. I thought that was a good little pro tip. So steering a long run in Claude Code, what to do, put a short rule in your Claude MD file about when to stop and ask and when to keep going. So you need to be proactive and tell it when to stop when you want it to. This is an example. When a step doesn't need my input, keep going. Put status notes in the same message as your next action. Stop and ask only when you can't continue without me, or before anything destructive, deleting data, force pushing, or anything outside this repository. Actually, one of the key things, one of the metas that I've been doing, this is skipping ahead a little bit. Okay. So, for the audience, if you have, if you're an AI builder and you're trying to ship a bunch of stuff, like promote it from staging to production, and you have to coordinate a whole release train of features, And maybe you're doing blue-green deploy, canary builds, maybe you're doing like feature flags and A/B experiments, stuff like that. I find myself in such a position, and one of the ways that I have tried to communicate the exhaustive tickets and pull requests and the complexity of the dependency graph to move from staging to production on such a feature train release, coordinating lots of cross-functional stakeholders, if you find yourself in that position as like an AIPM or AI engineer or developer or a builder. I have this meta of in Clarity Align, this is our little app I built for this, which is just to align our humans and agents on the same context. We use this HTML artifact spec to coordinate the release train. Here are the opportunity solution tree opportunities we're focused on, where we're shipping, the success metrics, and all of the, hey, here's where we're standing, the server's baking, the app's baking flags. decisions was making, how many PRs, what PRs, uh, what are the controls? All of this like engineering-oriented stuff up-leveled into an HTML visual artifact that as the agent is running its loop, you could see the latest was yesterday. It had an OTA update to the app. Uh, it actually goes in and updates this as it goes. So that's one of the ways that I've been using, uh, some of these latest AI tools to keep stakeholders, humans, and agents up to date with like a complex A release train. So anyway, just sharing that as like another meta to go into how do you steer a long run in Claude code and especially telling it, this article tells you stop when you want it to stop. That's great. But I'm also saying, how do you stay aligned with what the agent actually did? Give it clear instructions for when it finishes certain milestones in your autonomous goal loop, go back to some visual artifact, update that artifact. And for me, in this case, like I just have this all in one place so that my stakeholders understand it. I understand it and all my coding agents understand it. Cool. Jonathan, I'm actually curious from your CTO perspective, like thoughts on steering long runs in your experience using Claude Code or Codex?

Yeah, I was just looking at that in the documentation. I'd say historically that Codex responds a lot better to steering. And like, in fact, I think it's the intended way to use the model, whereas like Often trying to steer Claude Code has tripped it up and like, you know, kind of it hiccups as how it feels. So I haven't tried it much with Opus 5.5 yet. I've been, I like the input instructions, tell it what done is and tell it like when to stop. I've been experimenting a little bit more on that side of things, but the steering is something that I think they have some new features for and I wanna check it out.

Yeah, super cool. I also agree with you in Codex, you literally could type the thing and it goes into the queue and if you click steer and then it steers it, it like injects that. prompt mid-session to the agent. I don't know how they do that, but it feels great and it works really well, at least based on my own qualitative experience. So asking the agent to split big work across subagents. If you're an AI builder and you've got to parallelize a lot of agentic work, maybe coding work, you might want to, with Opus 5.5, coordinate parallel subagents on long audits and migrations. So here is an example of how. Audit every service in services, the directory for the retry bug in the linked issue. Give each service to its own subagent. When a subagent reports back, check its evidence before you accept it. Finish with one table, service affected, yes or no, and the evidence. And what this may do is go and split this up into different subagents. The subagents report back, check the evidence before acceptance, and then finishes with one table. So that, that's a simple way that you might split big work across subagents. Another interesting case here is code review. I know this is more of a niche thing, but one of the metas for every builder is ask a different model to code review the output of this model, like any of the implementation changes of the model you work with. And for Opus 5.5, this is an interesting one I always look at for every model release is how well does it code review. Last I checked with the meta, Cursor was still the best at catching bugs. But looking at this is interesting. One early tester said Opus 5.5 at its lowest effort caught more bugs. than Opus 5 at high effort with fewer false alarms. It also explains its changes in plain language, so its pull request descriptions are easier to review. Well, I mean, if that's actually the change is, hey, yo, we're just better than Opus 5, even at high effort, it more so speaks to how garbage Opus 5 was, even at high effort. I wouldn't have taken this one as a win, Anthropic. I'd love to see a benchmark on the actual code review skill across models, or a code review benchmark, I should say, across models. Okay. So that's a nothing burger. Okay. Moving along. There's a lot more in the Opus 5.5 release. Jonathan, from your CTO perspective, is there anything else the audience should concern themselves with when it comes to this latest Opus 5.5 release?

I don't think so. I mean, like, I think these docs are really good. Read the docs and see what people are doing on Twitter to experiment and then yeah, use it on your own work and see, you know, what it's good at. I think that's the best test. It's just like how well it helps you get your work done.

Yeah. And we could just see a quick glance here for all the AI builders looking at Opus 5.5 versus Opus 5. Input tokens, cheaper. Output tokens, cheaper. Cache read, faster. So cheaper, better, faster. What more could you ask for? And it seems to have higher agency. It's not telling me it can't do things. The last model, Opus 5, I always had to remind it, actually, dude, you could look at 1Password and pull this API key to do this thing. Actually, dude, you could totally go and use browser automation. Your creators created you and they had that in mind and it's a capability. There were so many things I had to do about that. So yeah, go give it a try, as Jonathan said, and don't take our word for it. Try Opus 5.5 and I think you'll be pleasantly surprised. We are for marketing, sales, and product and development use cases. Broad discovers a novel enzyme system with CRISPR-like repeats, September 23rd, 2026. This is a really interesting development as Dario came out and said a few weeks ago that, hey, we gotta wow the public with solving and curing cancer, not just talking about solving and curing cancer, but actually doing it. Since then, we've heard about things like the Moderna melanoma personalized cancer treatments, and now it looks like some more is coming out of their scientific arm here. So Claude led a discovery of a possible new gene editing mechanism. About 950 Claude agents searched a DNA database for 21 hours. Costing about 210 million tokens, screened 200,000+ enzymes and narrowed to 20 candidate systems. That's reporting from Anthropic. Human scientists, just to be clear, did all the wet lab work. This is a preprint. This isn't peer-reviewed yet, but it is showing more and more that, hey, the AI models, these frontier intelligence models are being put to task on human problems, like some of the mathematical theorems that are have been solved, are being solved currently. So Anthropic has not shown that ART can cut, copy, or paste DNA yet, but Dario came out and said his own take on this and, quote, this is work he would've been proud to do as a PhD student. This is Claude-led discovery of molecular machine that we suspect could represent a new gene editing mechanism. Its precise function, biotechnological utility, if any, or level of significance is not yet clear. But at minimum, it is work I would've been proud to do as a PhD student. The work was done mostly, though not entirely, by Claude. Our life sciences team suggested a broad area of research. Claude read through the literature and a bunch of genome data and discovered something interesting. Then Claude proposed experiments to verify discovery, and our team carried them out. So this is Claude acting as an orchestrator, the brain, and then the scientists in the wet lab being the hands. That's a very interesting meta, but I think we're going to start to see more and more of this, which is essentially Claude acting as a great PI, sensemaking in the world, all this data, then pointing out to where can we find more truth fundamentally. That's what I'm taking away from this here. Dario goes on to say, in 2023, models struggled to do math at the level of an average high school student. In 2024, they started to do well on math competitions for the best high schoolers in the country. In 2025, they started to solve minor open problems. In early 2026, more significant open problems. And in late 2026, they're beginning to solve the top few open problems in all of mathematics. We believe AI for biology is on a similar exponential trend. As Alex from the Moonshots podcast would say, all domains are cooked. Like math is cooked, science is cooked, biology's cooked. The AIs are here to stay and we're in the singularity, folks. Jonathan, what are your thoughts on this? latest life sciences work coming out of Anthropic? And what does this mean for frontier intelligence in general, if you could speculate and give your hot take?

Yeah, I, you know, I think it's really interesting. I'm definitely not an expert in this field. I do follow it, you know, casually, but it's hard for me to understand the significance of the breakthrough. I think what's really interesting here is that the model posed the hypothesis, it instructed the research, and I think that's something new. It's worth reporting on.

That's something new.

What I think, like, you know, in the Navier-Stokes and the series of problems that are solved there too, I have kind of like just some suspicions, right? And one is these problems are really complicated. Only a few people in the world can actually verify that it like did something right and that this is meaningful. And like this Navier-Stokes example, like I have suspicions that like maybe there's just some slop in the actual solution proof and like We don't have the capability to verify it yet.

How do we verify this? No human's going to verify the output of a ton of slop is slop, right?

And so I think they're sticking to domains where you can come up with a hypothesis in a highly verifiable domain. And I haven't read about how the wet labs were exactly used in conjunction, but like, I think, is this result reproducible by other labs? I think we're going to see some of that when it's peer-reviewed. So yeah, like really interesting. Is it actually solving meaningful problems that are going to like accelerate pharmaceuticals, extend life, cure cancer? That I'm more skeptical about, but it's really kind of like a cool direction.

Yeah, I really look at this and I'm like, the trend is AI will help more science go faster, more better science faster. I think that's sort of the trend here, but we're early, right? We haven't solved all of cancer. We haven't solved for world hunger. It's like there are many problems that we got to solve. So this is like the first of signals that I see that Sort of show me that the light at the end of the tunnel is there, but we should be skeptical that it's already here. So, uh, don't believe any headlines that it's already here, but stay optimistic that, hey, we're kind of going the right way. Uh, okay. So what are our twins' takes on this? This is interesting. So I'll go first. Let's do a twin check. Robert's twin. So first of all, the model came out and said confidence low, thin evidence, no recorded views on AI for scientific discovery or who gets access to it. The answer extrapolates from a general concern about lab dominance. So here's my twin's take real quick. I'm not a biologist, so I'll stay in my lane on the science. What I will say is that the humans still did every single lab step, and it's a preprint. So right now it's a faster search, not a new scientist. My gut is it's both. The method spreads to everybody, but the ones who could throw 950 agents and 210 million tokens at one question are the labs. So I lean towards an advantage for the biggest model owners, at least for now. And it stems from my worldviews and words from A July 17th podcast we did and an August 25th podcast we did. And this key belief 84, the AI labs may dominate the market, raising concerns about the protectability of economic value in AI. And belief 210, Robert believes the public has a negative view of AI, which is a significant problem for the industry. It is a problem for the industry, actually. So, Jonathan, do you want to read your twin, what it said here?

Let's see. Yeah, I think this is normal technology. It's a faster search through a lot of data. Humans did every lab step. The thing does Show me where the asymmetrical advantage is. A big run of agents over the public sequence database is a kind of method that gets copied and gets copied faster. And there are good open weight models, but I'd worry about the part I worry about is access because if this only works on the biggest proprietary models, then that's an argument for treating these models more like a public utility.

That's that. Yo, that sort of sounds like you actually.

Yeah, it definitely sounds like something that I would say it. I don't know if I'd say it in this. circumstance, but you know, it's not bad. It's not, it's like maybe, maybe I should have thought about it that way is kind of how I feel after reading it.

Thank you. Thank you, Jonathan's twin. We have more to think about now. Asymmetrical advantages or disadvantages in the science itself. That is an interesting thought. Hey, how did our twins do? I think my twin, let me grade it real quick. I liked that it said real quick, I'm not a biologist, so I'll stay in my lane on the science. And I do have this core value/belief that I express, which is I know what I know and I don't know what I don't know. And I want to not waste people's time. So I'll be very upfront about what I know and what I don't know. So I do like that it was very upfront about what I know and what I don't know. I'm not a biologist. I'll stay on my lane on the science. And I did actually, before my twin gave the take, my own take was saying that, hey, it's like the humans were the hands and they did the wet lab stuff. So they still got to do it, but the model made a lot of sense in the world and acted as a really good PI. And yeah, this is interesting. I don't think this is a divergence from my worldview, actually. I think this extrapolated pretty well given that we haven't talked about, or the self-model here, the twin here doesn't have any of our talks about scientific discovery at the intersection of AI. So I think the extrapolation Based on some of these other beliefs and episodes of my twin, did a pretty good job. I think the tone and the taste is still a little bit off. Like I wouldn't have said this. My gut is that it's both. I don't even know what that means in this context. And I might say it's a preprint, not a peer-reviewed study. So right now it's a faster search, not a peer-reviewed scientist, 24/7 scientist. I think that's a more effective way to say that. Cool. Just did an eval of my twin's output. I wouldn't have said this part, which is, so I lean toward an advantage for the biggest model owners. I don't think there's any sort of that here. I think it sort of misconstrued that in this context. Overall, I liked the, again, where it reasoned from. The extrapolation was quite good given that we haven't, my self-model didn't have any evidence of me talking about scientific discovery at the intersection of AI. I think it just missed in terms of my tone and how I would like sort of position the messaging and Sort of the questions that I would have to take away from the topic. But I thought pretty good take overall though. I would probably fail it and then throw in what I did as open coding. Jonathan, what about your twin's take on this, the asymmetric advantage take?

Well, I really liked the first line because like my, the cognitive immune system I've been developing, is this normal technology is actually at the top of the stack, you know, like, is it normal or is it asymmetric? So I like that it applied that. And, uh, yeah, I like, this is like, if given a few choices of how to apply, how to reply to the circumstance, I would consider this whole thing as it was typed, like, you know, a pretty good recommendation for how to think about it. Yeah, it's interesting. It put in like access, uh, and I, you know, this sort of wasn't crossing my radar, that vector of like, you know, what the story applied to. So I think that was a little bit of a stretch, but again, not bad for a topic that we don't have much like domain data on.

Yeah, I think the bigger story there is the who gets this capability of finding more truth and defining what truth is though. Like that, that could be a whole podcast on itself, right? It kind of goes into the micro and macro dialectics that is humanity's ever-evolving search for truth and meaning. That's just been our whole civilization. We can go pretty deep on that, but that's a, you know, that's a whole other thing. Instinct is a personal assistant that understands what you're working on and what's important to you. Connects to your applications and devices, email, messaging, screen, audio, location, and more. The interface is simple. There are no interfaces. It's trained to use a phone and a computer. You could text or call it. It uses its device in the same way that humans do. Instinct's core model is trained to understand and handle the deeply personal nuances of everyday life, following up on threads you've dropped, proactively calling or texting you, arranging a ride to the airport, booking a handyman for your apartment. And much more. Get started. This is pretty cool. Like a pretty cool way to onboard people is just text them, like text the model or the product. So Instinct, hang on, is this really their website?

$10 billion landing page.

This is their landing page. Are you serious? That's wild. So real quick, Instinct's founder says the viral AI agent's hallucination could spit out what it looks like a stranger's date.

Lots of security issues.

Yeah. Lots of personal AI agent horror stories. I'm just like going off of the top. I think I found an AI agent worth the risk. Instinct saved me $550, booked my restaurant reservation, and warned me about a phishing scam. It also wasted $64 and might be a security nightmare. This is from Wired. You could read more about it here, but it looks like a $2.5— I think it's a $2 billion valuation company. And it more speaks to this thing I wanted Jonathan's take on. Which is we're seeing Muse come out from Meta. That's number 1 on the app stores last I checked this week. We're seeing GrokBot come out. That was just most recently, I think, top 3 on the app store before Muse came out. And now we're seeing these other startups kind of fighting for wedges and beachheads, such as Instinct for the personalized AI agent game. We're also seeing incumbents In this market, if we can call it that, where it's such an emerging market that started with OpenCLAW and then fast followed by, you know, like IronCLAW or Hermes Agent that tried to solve for this form factor. And now we're seeing some of these bigger companies and more well-funded startups like Instinct go after this personalized agent beachhead. So I'm kind of curious, Jonathan, from your founder lens, what are your thoughts on personalized agents and this like B2C focus that we're seeing come out of Facebook, come out of SpaceX AI and companies like Instinct?

Yeah, I think another name to lump in there is Town. Town's also an AI agent that's got a ton of traction alongside Instinct, Town AI. Yeah, so like there's been like kind of a renaissance of consumer personal agents. And I think that's the first thing is that these are consumer products. And so the success that we're seeing with Claude Code and with Codex, which is eating up most of the tokens historically, Now there's user experience coming to consumers. The Meta Muse was purposely like modeled off of OpenCLAW. And I think Meta Muse has the biggest distribution platform. It popped up Meta's stock price when it went out because it was so efficacious and good. And I think like, if you think that Meta's got a strong advertising business, they have one of the best advertising businesses in the world, them and Google. Now you can have a shopping assistant just Buy what you want. And you like, so instead of an ad that's contextually relevant to you, it's just like, you know, here, I'll just buy this thing for you.

Right.

And I think the big news this week was that Amazon blocked Metamuse and they said like, well, per our terms of service, you can't have an agent buy anything on our platform. And they seem like they're, they're going to hold up that line. So whereas like Shopify did the opposite, Expedia did the opposite. And they said like, we're on the Metamuse, like, yeah, buy shit from our sites. And I think there's a question of like, where, whether Walmart will join on to Meta Muse also, and whether they'll have like an adequate, like, you know, shopping inventory. So I think, yeah, AI-assisted shopping is a big use case. I think people are using this for calendars, for email assistant, for generally managing like anything in your personal data. They all have like different capabilities. If we jump over to Instinct for a second, I think one of the things they did really well that I hear, I haven't really used it a lot myself, is the memory structure. And we had a nice article about like how Instinct built the memory structure. And it kind of looks like this, um, like the wiki system that Andrej Karpathy described, where it was a bunch of Markdown files being managed by a Git system. So a version set of Markdown files.

Real quick, if people want to know what a $2.5 billion valuation startup memory architecture looks like, this is what it looks like. Conversation messages come in, current conversation context. Background processing, that's slightly a black box. That's the secret sauce IP. Accessible Markdown files, just like OpenCLaw. One-pager generation, mechanism unknown, also black box, probably secret sauce IP. Search and reads, so your normal CRUDs and stuff like that with indices on demand. Those probably solve problems. Identity profile plus to-do index plus compassion recap. And that all feeds into the while loop that is the agent's answer.

Right.

So. Yeah, this is the reverse engineering Instinct's memory. How does it actually work? Instinct keeps its memory as git-tracked Markdown files found with grep rather than vectors. Here's the whole system as far as black box probing can reconstruct it. How do we build our own super memory in 16 lines? This came out of the Super Memory blog. They're all about memory. It's a problem space that isn't solved yet. We're really interested in it. as well in modeling the self, digital twins. That's what we do. So obviously memory is a big part of what makes us who we are and what drives decision-making, what drives taste, what drives subjective sense-making and perception. Jonathan, sorry to interrupt. I just wanted to share this article and then show that part of the memory architecture. Do you have more to say on this?

Yeah, it's a great article. You can go through the specifics of the type of memory that it stores, but I think this is what it all comes down to. You know, like whether if you, if you're on Meta's platform, they have access to the data that you have on Meta and then in other places you're giving access to, how are they going to shape the memory to bring in-context judgment, you know, the in-context information that you need to make judgments that are actually helpful to you? And, um, so Meta's got kind of an advantage there. If Google makes something like this, they're going to have the Google set of data and they're not going to share that with anybody else. Right. And then like on your device, I think, you know, maybe use something like Instinct. You can probably integrate as many sources as the APIs allow. And I think like, you know, if you're building something, this is like, this is the lens that you want to use to understand like what brings a great user experience is like, how do, how is context engineering done? How is the memory shaped? And how is relevant information brought into context when the user needs it? Right. So I think that all these, all these products that, that consumers are using are really like good learning lessons. This is just first generation, by the way. So these things are going to continue to evolve. And as an AI builder, you can build your own, you can build them for other people, you know, you can keep evolving the field and evolving, like, you know, how memory and context might be delivered. But the big tech companies have the advantage of the data they own and they don't let, like, let you use it as developers. So there's that.

Yeah, definitely. So real quick, there's this concept of profile. We have a similar concept on our self-models and the usage of them on judging agent runs to compound Your private evals as a company and a team. So for a profile, that's how we tend to route the right context at the right time. That's part of our design and architecture. For them, profile is essentially just what the model always needs to know about the user. So like a user profile or like a gamer profile might be a way to think about it. It's a personalized, you know, what contextualizes this ask that the user has, this query that the user has. So like life context, A summary of selected user circumstances and relevant people at work. Autonomy calibration, selected preferences about when the assistant should act or ask. Channel communication style, selected preferences about how to communicate. So those are the headers that Instinct sees. And then profiles are formed like this. So you have your entities/people, entities/orgs, knowledge/facts, knowledge/preferences, knowledge/decisions. Com/phone, timeline/daily, timeline/weekly, workstreams/active, workstreams/completed. So this is the file structure that Instinct uses after a few days of trying to reverse engineer the black box. Yeah, we could— you could go pretty deep on this, but you could see, you know, the experience of what makes a personalized or hyper-personalized agent comes down to, does this thing remember me? Does it treat me differently? And one thing I wanted to say, actually taking a step back out of this, out of the architecture and the technical details of context engineering a little bit, is I think there's this like first order story, which is personal agents have a market. That's the first order story, which is all we're seeing a lot of competition go after the business to consumer, use my AI agent more for your everyday things. The second order story to me is that one about Amazon disallowing MetaMuse to interact with Amazon, but Shopify and then maybe Walmart fast following, maybe allowing MetaMuse and other agents along with the humans to make economic choices in their ecosystem. I think that's the more interesting story, which is we are now seeing an entity besides humans decide the deploy of capital. In our markets. That's such an interesting thing to me, right? Like, what does that mean for the future? Like, now, now we're seeing it because before we saw it at Cloudflare, Cloudflare was like, hey, we're going to allow for agentic commerce. And now it's like, hey, agentic commerce is here. It's not just allowed for, it's, hey, a really big company that has billions of monthly active users Facebook is allowing people and their agents through Meta Muse to, in an agent-first economy. I think that's really interesting. It then goes into, okay, what class of problems do I have as a human that I would ship off to an agent? Or like, how much money do I trust to an agent? How much can I afford to trust to an agent? Right? It kind of gets there now that we get into like economic classes of use cases versus help me decide things, help me aggregate data, help me find the right thing to buy. Now it's like you haven't just helped me find the right thing to buy, which is erasing some market friction between supply and demand in the economy. Now it's, you're helping me actually deploy the capital, exchange the hands, exchange the money. Yeah, I think that's, we're going to look back at this time in a few years and be like, wait, what the fuck? That's crazy. Jonathan, do you have any thoughts about that? Or am I like just on an island freaking out here about just potential implications of agentic commerce? meeting the business-to-consumer landscape?

Well, yeah, I think it's important. It changes the user interface and the relationship between the company and the consumer, you know, and I've really enjoyed Ben Thompson's aggregation theory where he talks about like aggregation of attention as being like the scarce resource that is driving the economy right now. And it's very predictive in terms of how we can understand how these tech companies evolve. And here you have like a fundamental disruption in how human attention meet the interface for working with Facebook or Google or Amazon or whoever else. Yeah, this seems significant. I don't know exactly how it's going to play out. And I think another mental model I have is that we're watching all these big tech companies become AI monoliths from semiconductors to the consumer interface. It's going to be OpenAI and Anthropic and Nvidia and Amazon, Microsoft, Google, Apple, and Meta. Right. So all of those companies are kind of transforming into the same shape. And now we're getting some like details of like, how's the surface area going to kind of be refined between the user interface and these companies? Um, and, you know, like, uh, it's just really interesting to see the shaping forces, especially as they move so fast.

Yeah, it is really interesting. Like what happens when, say, your muse buys something? And then let's say the purchase gets messed up in some way, or they purchased the wrong thing. You know, like what are those, what are those, what happens? I think there's going to be landmark cases the next few years around this agentic commerce that's meeting the world by storm here. And we're going to have a lot more precedence that needs to be formed for what happens when things go out of hand or not expected based on the human and their intended outcome that they put into this little prompt box. to this all-knowing agentic being that should understand the human's goals, right? I'm sure that will always go right. It's like, no, dude, it's going to go wrong. So I'm just, I think this is a very significant moment that we're going to look back on and be like, oh my gosh, economists will look at this time and be like, this is the first time ever that a non-human entity made decisions on commerce and economic activity. I think that's going to be crazy. When we look at the charts in a few years. Yeah. Okay. So that's personalized agents and instincts. How do you even make personalized agents good? Memory systems, memory systems, memory systems. Okay. Another thing that came out that I am really excited about because we have this use case as well is diarization. So NeboTron 3 came out and it's specific from NVIDIA. This is an open weight speaker diarization model. Designed to determine who spoke when in real-world audio. It supports both streaming and offline inference and handles up to 8 speakers. Following SortFormer, the model resolves speaker permutation by ordering its output channels according to each speaker's first arrival in the input audio. So you can see here what a little demo of this would look like for up to 8 people talking. It can detect who's talking and in real time be able to get the diarization To an efficacious degree, why does this matter for digital twins? Attributing and building up somebody's model of self to us, we hold the philosophy that it you know there's different levels of evidence that would inform a probabilistic confidence model of what your digital twin says about you, how it models you. And one of the key use cases is absorbing in a meeting transcript. As we're all in these AI recorded meetings nowadays, we can get down to who said what from what meeting on what day at what time. Why? And what was the context? For our work, our company brain work, even at our own startup, uh, Clarity, uh, that I, that I work on with Jonathan, like we're always uploading transcripts. Like we're, we're dealing with transcripts all the time. Like we leave a meeting, a transcript is one of the first, it's like a first citizen, first class asset to our company. So anyway, I, I was curious about this 'cause we've tried so many different solutions. And so I threw this in the team learnings chat. I saw this come out. We're going to have to give this a shot at some point. Not too much to say here other than the benchmarks are really good. And this is a free open weight model from NVIDIA really investing into the open source, open weight community game after they bought Hugging Face. So really cool to see. There's so much more, but we got to build a company. We don't have time. So I'm going to prioritize. So Arise, Arise has a great blog. I would recommend people sign up for their blog. Typesafes JED. So Jev is the latest hotness, 440 times cheaper, 109 times faster than the competition with their System 1 models, which introduces a new class of models. One of the use cases is LM judges. So can TypeSafe's Jev decision models replace LM judges? TypeSafe's Jev classifies scores and routes without generating text, up to hundreds of times cheaper than an LM judge. What that changes for evals, conference routing, In application architecture. So, JEV versus LLMs, early benchmarks on accuracy, cost, and latency. If you're an AI builder and you are trying to figure out your evals loop for closed autonomous— closed-loop autonomous agents, you want to listen up here. Everybody wants to get to this amazing LLM judge for some domain-specific workflow, for some domain-specific harness that will eat up everything. And the more data you put into it, the more of a moat and IP it builds and accrues. But You know, that, that there's, there's the promise and the idealism, and then there's like reality. So I love this, which is comparing the use case with Jev with their structured outputs and free output tokens and much cheaper and faster application to this particular workflow. So they published evals for decision workflows, one of which is reviewing a finished agent trace to decide whether a human needs to look at it. Averaged across the 4 workflows, Jev lands at 68% accuracy at 0.0004. Or dollars and 0.4 seconds per case. GPT-5.6 Terra is at 68% for 3 cents. So that is a couple of orders of magnitude difference and then 10 seconds. So a couple orders of magnitude difference on both cost and speed. Opus 5 is at 73% for 18 cents and 38 seconds. So that's 5 points behind Opus 5, which is the leader, but on the other hand, it's 440x cheaper. That's a very interesting cost-benefit trade-off, such a radical one that may change how we architect applications. So this article goes into a little bit of why did LLM as a judge become a default for AI evaluation? Why are you using LLMs to make decisions in the first place? The reason is simple. They are able to do it without huge, expensive training sets, which is what most ML solutions prior to LLMs required. So here are some options. Approach: logistic regression on embeddings, fine-tuned encoder, Frost encoder, we ranked small team judge, Frontier LLM as judge, decision model Jeff. So here's a little cost table on the different approaches to make structured decisions in your ML pipeline and eval pipeline. And you can really boil down to a few heuristics. Do you need labels? Cost per decision, latency, which affects user experience, probability you can act on it, and then explanation required or not. So for your AI Workflow, LLM-based product workflows, you should ask about these things. Jev is down here, the newest entrant to maybe a forming market here, but you could see they outperform everything in terms of latency and cost here. So it's definitely a strong tool in the toolbox to consider. Other things to skim through, then I'll pass it to Jonathan for his take. Zero-shot classification with Jev. Decisions without task-specific training. It takes the same plain English criteria you put in a judge prompt, needs no labels and returns a probability that actually could, has reached 98.3% accuracy without task-specific fine-tuning or labeled examples in its requests, which is really promising. What might you lose when you replace an LLM judge with Jeff? The biggest loss is the explanation of the reasoning. TypeSafe's docs say plainly that System 1 models don't generate explanations of their reasoning. In NearHere's test noted the same thing. A category and probabilities came back, nothing else. Depending on your use case, that could matter a lot. LM judge explanations can be incredibly valuable. That tells you not just what was wrong, but why. That provides real signal en masse back to a coding agent to automatically improve the software. JEP just gives you a probability. It's much less directional. Um, but what Arise recommends is actually a meta for AI builders, which is why not both? 4K, no loss in dose. Why not that, right? So all traces, this is the architecture they're kind of recommending, use both for cheap coverage and explanation, expensive explanation. So all traces, every run, every tool call or agent step you afford, you can afford to score, throw that into JEV. It's 440x cheaper. Label and probability assignment, broad measurement, no written rationale. That's what JEV does. It categorizes into confidence cases, so it keeps your score, wanders, drifts, No extra spend, and then rerun a sample through an LLM judge for this red area or failures or gray zone scores. And then that should go into overall a coding agent or a human plus coding agent to fix and iterate on. So the TLDR meta for AI builders with, hey, I have an evals pipeline and I have LLM judges. What do I do now that Jev has come out? The recommendation from Arise here is do both. Jev is really cheap. And it's a great classifier. So for the classification use case, have it on every trace for 100% coverage. And then for things that you're unsure about, like sample failures, uncertain scores, gray zone kind of areas where you want to bring in an expert to annotate, or you want to bring in LLM Judge where you use more tokens from their frontier intelligence and you need a written explanation to improve the system, bring that in. So this is, this is like a recommended use case for them here. Another view of the same, and then I'll pass it to Jonathan for a CC. his CTO perspective. So today's default is pass/fail, route A/B, pick and label, Frontier LLM judge. Split stack would be typed questions, decision model goes to classify score route version, swap this layer on its own. And then text that's got to be written, LLM, that goes through the LLM for tokens, for summaries, replies, explanations, reasoning. And then you pay for generation from there. Because again, Jev has no cost on output tokens. LLMs do. So that, that really is a huge benefit, new tool in the toolbox for AI builders to improve their pipelines from a cost efficiency and speed standpoint. Jonathan, what are your thoughts from your CTO and founder perspective on Typesafe's Jev with this LLM judge meta?

Right. I think it's, I think it's changed the game. I mean, I think in doing evals and using LLM judges, like you're just, you're going to benefit from using Jev alongside the existing process. Um, I actually, uh, in this Rise article, I had, I had like researched with ChatGPT, how would Typesafe recommend using Jev for evaluations and as an LLM judge? And, uh, you know, I think like I'm using it right now to actually like predict how expert labelers might label certain data. So I'll generate synthetic data with synthetic labels. And then as we actually get the real labels from the experts, from our clients, uh, I'll calibrate. Right. The, uh, the JEV models. And so we sort of like start with a judge that gets calibrated as we go along. And it's just like, it's just following along for the ride. You have a JEV representation of the evals process the whole way through. And I think that's the System 1 model kind of like in action is that you're applying System 1 throughout the evals pipeline. And one surprising thing to me when I did the research was that not only can you build an LLM judge, But you can actually use Jev to improve your explicit domain models. So if you're making a domain model of a product manager or a writing checker—

Compliance expert or privacy expert or something like that.

Sure. Yeah. Any of these cases, like you can actually use Jev to do checks for the inputs that you get and round out the cases that like, say, because if you have Jev evaluate, you know, a given document and you have an expert evaluate it, there's going to be differences. And evaluating those differences actually rounds out the robustness of how explicit your domain model is. And that's one of the challenges you have when you're labeling. It's just like, how deep do I go? Exactly what do I mean? And if you can get it to the point where a model like JEV is consistent with the way that your experts are annotating, you have a domain model that is going to be like, you know, and you can get it more reliable.

quickly. That's the key part is like going from zero to reliable could be faster now with this tool choice. Is that what you're saying, Jonathan?

Yeah, it's kind of like having the Copilot or autocomplete alongside you as you go. Um, so yeah, I'm just getting started using it, but I think like in our case, generating both synthetic data, synthetic labels, carrying the JEV corpus through to create an LM judge, and then using it to stress test the explicit domain model that we're creating, it just feels like Our eval process got upgraded. Like, that's a win. Yeah, we'll see how it works out as we go through this next cycle with our clients.

Very cool. Yeah. So for any AI builder out there that is struggling either with a greenfield AI application or brownfield, you've gotten out there, it's a good demo, but your execs say different user queries than you expect in the demo, or your customers do, and you're responsible for objective, measurable, incremental improvements sprint over sprint for your LM-based products or workflows, look to Jev to upgrade your evals pipeline. Also look to Hamel Hussein's work in evals. Uh, couldn't recommend that guy's stuff enough. We're about to do some guest podcasts with him. So, uh, look out for that in the next few podcasts. We're going to do deep dives on how to build, uh, objectively measurably improving agent products that are going to be better than the competition, regardless of domain. Some of these methodologies, you won't want to miss out. So this is a cool new tool that came out and it's now in every builder's toolbox. If you can get onto the waitlist, I hear that the founder slash developers on the founding team are friendly if you join their Discord. So feel free to do that. And it, they do seem like a cool team. It's led by this guy, Diogo, who came from OpenAI and wanted to, had this vision for System 1 models to help drive. He said this one line that we referenced in the last pod, go watch his video for the marketing launch, but he said, we build for prod, not— we build prod, not God. And he's really focused on like, let's get real production engineering use cases enabled and empowered. And he's done a great job here getting Jeb out the door. Very cool. Okay. I think the last thing we'll go into actually is just our own twins takes on this, and then we'll save the recursive self-improvement paper for next time. So, I'm gonna read my twin take. Robert's take on this Arise Jev article. Yeah, so for narrow, high-volume jobs, if it matches your human labels on a held-out set, that's enough. At $0.0004 a case versus $0.18 a case, I would run it on every trace. Where I draw the line is anything like hiring, where a number with no reason can hide bias and you can't question it. So use it for coverage, then send the failures to a judge that explains itself, which is basically Arise's The trust comes from your labels, not from the judge being able to talk. Overall, pretty good take, I'd say. Yeah, actually, I love this, that it brought up that, hey, there's this use case you've talked about in the past, me, around hiring, where I've even worked on that problem before for like agentic resume parsing. That's a very tricky domain because there's compliance in every different geo or region around the world around that type of thing. And that's where bias and Um, you know, like real-world harm could occur where it just slips under the radar. But yeah, I like that my digital twin was like, okay, yeah, look at this. So you should, you should have a reason for sensitive areas like hiring where bias and what I'm seeing and potential harm in the world could happen. I think that's good. And that augments my original take, which it is in line with, which is use it for 100% coverage and then use an LN judge for like gray areas. It didn't quite say gray area. Failures or gray areas is anything like hiring, which can have potential personal harm. Cool. Yeah. Overall, I like the take. I'd call it a pass just with some tone differences. And it did remind me of one of my personal values, which is like, think about the personal harm or bias involved with a case like this. Pretty cool. Good job, Robert, digging this one. Jonathan, want to read yours?

Yeah. I mean, I was pretty excited about Jev and what I wanted for asking a whole bunch of questions against a corpus fast and cheap. And I think you can question it. You question it differently. You ask the same question with different choices and permutations, and you see if the answer holds up statistically. So if it matches your human labels on a held-out set and survives this kind of probing, then yeah, I call that an eval. I don't need pros for every trace. You need the pros for a sample of failures, right?

Interesting. What do you think about What your twin said, Jonathan?

It seemed like kind of an incomplete, like a slice of a take, you know what I mean? It was picking up on the choice permutations paper that we talked about last time, which is actually not a bad technique to use for the synthetic labels I was talking about earlier. So that's kind of cool. Yeah, but it kind of went down an analytic rabbit hole instead of looking at it as holistically as I would normally try to do.

Yeah, I see. Like it sort of went too deep in the depth bucket and not enough breadth in the breadth bucket. That's what I'm seeing here.

Yeah.

Yeah.

And I mean, like, I think we just talked about Jeff so much recently, it's probably like a little overweighted, you know, uh, not open weighted, but overweighted.

Only another AI nerd would get that bullshit that I just said. Thank you, audience, for dealing with us. If you like corny dad jokes in the AI builder realm, Come back to Weekly AI Clarity. Anyway, okay, just moving us along. We are gonna, this was fun to do our digital twins and the evals live. So with that, I'd say my twin got partly right. I would call it a pass in terms of reasoning, taste, changes. I did some open coding just there. Jonathan's, it went too deep and went down a technical kind of like rabbit hole, didn't go broad or strategic enough. So call that a fail. And our twins will get better from here. We have an RRSI paper from Google that we haven't been able to go into on this podcast because we've already gone long. There's been a lot to cover this past week in AI world, and we got to get back to building our company. So I'm going to stop the pod there and just want to say, for those listening on Apple Podcasts, Spotify, YouTube, wherever you're listening from, if you have comments, if you have Meta, if you have news, if you have Memes that you want to share with us, slash me. I handle all the podcast script and comms. So hit me up on any of our socials. Right now I'm checking Instagram the most. So we're posting reels and shorts every day pointed to the long-form podcast. And we're really doing this for you all. We love doing this. It's fun to get together and go through our own takes of the latest news, memes, and meta for AI builders. And we love the community that we're starting to see form around this work. So definitely keep supporting, definitely keep coming out, make comments on all the videos and stuff like that. And I'll be watching those, especially on Instagram and on YouTube. If you have stuff you want us to share on the podcast, please go ahead and send those in and I'll scour those channels for that. Anyway, this has been your weekly AI clarity, your signal from the noise of the AI news hype cycles. Happening every week, coming to you on Spotify, Apple, YouTube, wherever you want to listen to your podcasts. Hope you have a great day. This has been awesome. Take care. Bye-bye.