Reasoning Models & LLM-as-a-Judge

So today we have Alex Volkov. He, you might already know him. He runs a very great weekly news newsletter and podcast called Thursday AI. Thursday AI. How do I pronounce this?

Thursday.

Thursday AI. Yes. It's a really good kind of stream of news. It's like very fast, high intensity. Really good, one of the best ones out there. Alex also works at Weights and Biases. He teaches people about evals also. And, you know, he's like been in this space. Alex gives the best talks. So like, I believe that you should educate and entertain at the same time. And Alex is like the epitome of the fusion of education and entertainment. You should, and so you should go see, you should Google his name and look for LLM Judge Talk, where he dresses up like a judge and he judges various things. And it's one of the most fantastic talks out there of its kind. And okay, so with that, I'm going to hand it over to Alex.

Thank you, Hamel Husain. Thank you for all that love. Hey everyone, thank you for joining this. Hamel set me up for success here, and I'm really hoping to kind of follow up and actually give you an entertaining talk. Though this topic is very different than the previous one, and I'm not wearing any costumes for this one, but maybe for the next course. I've been following the course a little bit, and it looks incredible. So some of the stuff that I will say, kind of, we'll just— you guys have already went over, but I'm hoping to also kind of bring you some some new and interesting things here as well. So let me get started and go right into it. Let's switch this. Hopefully you guys see my slide and not my, like, uh, yes, I can see it.

Yeah, right, it's already entertaining.

Uh, all right folks, this is basically the title of my talk, uh, and this is the thing that I wanted to talk to you about today, are reasoning models Better LLM Judges. Reasoning models, aka, also known as LRMs, which is kind of the latest thing in literature. They call them LRMs. And I started this talk, I started this kind of interest of mine, research, back in the maybe February of this year when reasoning was just fairly new to us. There was like a preview, O1 released around December, right? Those are like 12 Days of Christmas for OpenAI. Um, they also announced L3 is coming, and then Deepseek R1 basically with reasoning broke through the barrier of kind of the AI bubble. Everybody was talking about Deepseek R1, and reasoning was a big part of it as well. And then I gave this talk at the AI Engineer Summit in New York. Some of you maybe have been there, and it got a lot of interest. And when I chatted with Hamel Husain of How Can I help support this course and what can I bring to the table, I decided to give you the talk as well. And then since then, many, many things changed. And so the talk is updated until you'll see some very, very much up-to-date stuff. So why am I even interested in this? Who am I and why am I interested in this? Hamel gave me a little bit of a preview. I'll tell you about myself as well. My name is Alex Volkov. My official title is Senior Manager, AI Evangelist at Weights Biases. I chose the AI evangelist part because I do promote AI both internally and externally, and I do it in a positive way. It's very important for me to counter the doomerism that exists everywhere on social media. I don't like the fact that algorithms pick up on negativity, and unfortunately, when you go negative, the algorithm rewards you with likes and everything. So I really try to do things with a positive mindset as well. I'm like Hamel Husain, I'm working at Weights Biases. We have a product called W&B Weave. I'll show you a brief very brief example of it later on. And I'm focused on LLM evals as well. And the folks who run this course are a constant source of inspiration and content for me as well. So we've been talking about Shreyas' papers multiple times. Shreyas, I think you've been in Weights and Biases multiple times as well. Hamel Husain as well, you helped multiple things. And so I draw my inspiration and content from the folks who bring you some of the things in this course. And I will cite another guest speaker, I think Hamel Husain tomorrow or something, or a couple of days.

Yeah, very soon. Yeah.

Mention Eugene Yan, one great dude as well. I'm Altrain on X. You, well, feel free to follow me for the latest and greatest AI news. I really try to reduce the amount of spam and just give you a signal as the host of Thursday Eye as well. So we go live every week. We're going to go live this Thursday and we talk about what happened in the world of AI to kind of the most interesting news. Our motto is we stay up to date so you don't have to.

And I think it's one of the highest signal per token podcasts.

I really appreciate this feedback. I think it means a lot to me, uh, that, uh, folks are enjoying this. And I think, uh, one of the things to connect it to what we're talking about here, right, one of the things we always cover is new models, right? So I can give you a taste of it today. Mistral— you guys were busy looking through the course and maybe some of you haven't checked the news— Mistral released a thinking model today called Megistral. And we're definitely going to cover this on Thursday. And by covering this, we basically mean, hey, this model came out, this is their benchmark scores, etc. And this is what we think about this model. So we do vibe checks, etc. Sometimes we do more than vibe checks. And how this affects businesses and how this affects people who build with these models is very important. Right. And this is kind of the area of evaluations. This is why we This is why we do the thing. We need to evaluate and whether or not we can drop and replace, you know, Magistral into other model that was used for your production application. We don't know this without evals, right? This is like what the whole course is about. So I'm very passionate about evals specifically because I also do it every week. I'm also a speaker and a workshop instructor on LLM evals. And some of this content that you'll see is based from there. And so the rise of reasoning models, when I first wrote this talk, you'll see a few references of mine. I actually wasn't sure if to reference the previous version of the talk or not. Sometimes it's good to give a clean slate, but I ended up deciding I will reference because the, you know, we also do evals because the field moves so fast. We need some ground truth to Anchor 2. And so I do want to reference the fact that since I wrote this talk, all of these thinking models started dropping. Like all of these models have thinking, reasoning enabled with them. And now, as far as I saw in literature, the— at least the researchers refer to LLMs, which models without thinking, and LRMs, large reasoning models. And when you have a combined model like Gemini, Gemini, for example, 2.5 Pro is a combined model. Those are still considered LRMs if the thinking is turned on. And we're going to talk about thinking budgets and, um, as well. So this is my standard slide. So you guys know this quick evaluations primer. You compile datasets, you have model answer each of the tests in the datasets, and then you score or grade them. You have, uh, programmatic scoring, you have human loop scoring, and LLM as a judge scoring. And this is the one that we're going to focus on. Very briefly skipping this, thus the slides will have less time just because this is the whole course. This is what it's about. Honorable mention for evals and vibes, but we don't want to do evals and vibes. Vibes are not— well, we do want it, but vibes are not— you can't subtract the vibes of yesterday from the vibes of today to get an empiric score. That's what I like to say, right? That's why vibes evals are great, but not necessarily sufficient. I do want to highlight this. This is kind of maybe again redundant for the course work, but this is something that I kind of like outlined for ourselves and I really want to kind of highlight where we're going to talk about this. Here is basically in one slide the essence of all my workshops on evaluations, right? So it starts with Hamel Husain and everybody else's favorite, look at your data on the left and then continues to evaluate constantly on the right, right? So I think the most important important thing in this 1-2 framework basically is compilation of datasets. You would look at your data, but not only looking from looking at your data, you need to figure out what works, what doesn't work, and then drop this into datasets and you need to compile multiple datasets for evaluation and kind of create those. And then step number 2 is evaluation constantly. Constant is the highest, like most important part here, right? So offline evaluations, you should have them for major features. Online evaluations, you should be able to running like on a subset of traces in production, you should be having monitoring, making sure that, you know, generated things don't break. And then you should also have guardrails. And then having both 1 and 2 allows you to do this arrow at the bottom, confidently iterate, change your prompts, change your models, change your product features. So this is just generally a slide that I didn't want to skip. Let's get to the meat of the presentation, right? I don't know if you'll notice the little Easter egg in the slide, but since the release of 0.1 and DeepSeq and Qwen and like all of these things, we've seen the rise of test time compute. Basically what we consider is another scaling law, essentially. I will mention the Apple paper. Hold on to your hats. I will mention the Apple paper that came out just this week as well, briefly. And I would love to maybe have a conversation about this later. But I really like since the release, I started wondering, and then doing some research whether or not there was already research. I really wanted to see if researchers already approached this new paradigm of test-time compute, of giving the model more time to reason about things and apply them to LLM as a judge tasks and use cases, right? And this was quite a while ago where O1 was just like announced. I think it was, I wanna say October last year and the full launch was December last year. And since then we got O3. Was launched, O4 Mini. O3 Pro is launching today, supposedly at some point, you know. So we're like, we're moving forward with this paradigm of thinking. And, you know, many, many other models. I really want to see if this also helps us with our LLMs to judge tasks. That was the topic of my research.

It's like we're getting a little bit of Tuesday eye.

We're getting a little bit. Yeah, today was crazy, man. I just can't help it, as you see. And okay, so here's my hypothesis. LLMs can be better LLM judges. Okay, this is my hypothesis. We start with a hypothesis. I'm not a scientist necessarily, but I wanted to approach this from this perspective. And Hamel Husain, I think you pointed out in the course documents, and always like we talk about this also in workshops as well, you maybe want to start, and this is a suggestion, with the most capable model as you can afford to for judging, especially for offline cases, because they're offline and maybe you can afford to wait, whereas in production you may not be able to afford to wait. And, you know, latency is important for offline evaluations. Latency is maybe less crazy. And so you want to generally start with like the most capable model for LM's judge to give you the best results. And LLMs, when they came out, the highlight of reasoning was around math and code, right? These models came out And the main benchmarks that they were great on were the math competitions, AIME, right? Some Math Olympiad competitions and code, like Codeforces and different things like this, right? So very, very verifiable domains. These models were trained on verifiable rewards with reinforcement learning, and thus they really are more capable in specific areas like math and code and other verifiable domains. In addition, we can scale reasoning. So because again, our offline evals can run for longer, we can scale with the reasoning, with the test-time compute thing that they offer, and thus potentially giving us— potentially, yes, we're going to get to whether or not this works— potentially giving us significantly better performance on the judging. Some reasoning models, as we saw, if you joined the previous session, I saw Ashraya kind of looking into reasoning inside the cursor as well. Some reasoning models return chain of thought, which is they were trained with chain of thought built in, and some of them actually return this. So if you use DeepSeq R1, for example, or Qwen 2.5, I believe the open source models for sure, the providers will stream you the actual reasoning tokens and the response. Gemini 2.5 recently started showing reasoning summaries as well. Technically, while we— the standard practice, I believe, for LLM as a judge to ask it for its reasoning, it helps the model anyway. You can debug your judges based on the reasoning stuff. So OpenAI doesn't return any reasoning, even the summarized one, unfortunately. Maybe they'll switch at some point. But basically, this is the hypothesis. Because we could see how the model thinks before it actually responds, we can help understand bias, for example, for this model. We can maybe fine-tune our prompts a little bit. Better, right? I will pause on this point and say I'm very, very salty with Gemini specifically. In AI Studio, you used to be able to see the full tracing, and they switched that to reasoning summaries. And I'm very salty about this. And I've told everybody in Google who was willing to hear, including live on stage to Logan and Jack Ray, like, hey folks, please bring back full reasoning. It was instrumental to building the right prompts when you see how the model thinks before it responds. And the same for LLM judges as well.

Do you think they took that away because of distillation concerns and things like that?

I don't know, man. I've talked to them about this. They cited multiple other things, including whether or not consumers wanted, whether or not this was only for developers. But the feedback was given loud and clear, and hopefully, hopefully they'll bring it back. Distillation concerns could be one reason. Yeah. And these models, the reasoning models can self-correct as well. Like we've seen this. In multiple models, the, the upcoming versions of models— this is not in my talk, I'm just like ad-libbing here— but, but the upcoming API versions of models, O3, could, could do tool use in the thinking process, which is significant. So right now, if you go to O3 and you ask it for some stuff, it can go and like search up, you know, things for you while it does reasoning. It can use tools like image zoom in and different things, right? So technically, these models are going to come to the API as well with the ability to use tools with the thinking process. So this could definitely lead to better judgment, especially if you can provide those tools, right? If you want to provide your database tool and you want to see— you want to judge output of retrieval, for example, if you have a judge that can use tools in the reasoning, could definitely, definitely significantly improve performance in hypothesis, right? Let me see if I covered everything. Yes. So this is the hypothesis. But what are we actually looking to improve? Right. So this is kind of— there's a hypothesis. And then like, what are we trying to see whether or not it's better on? So as documented and probably talked about in this course like multiple times already, like all LLMs have biases. And specifically, there's outlined multiple biases for LLM judges. Right. This is citing the survey on LLM as a judge from Archive from January 25th. We have stuff like position bias where models prefer some position on the answers. If you give them like 4 answers, they could prefer the last one, for example, the first one. Verbosity bias is, and this is for LLMs, not necessarily LRMs. This is like a regular LLM as a judge biases. Verbosity, that models prefer longer responses, sometimes overvalue them even though the actual factual stuff within the models may, within the long responses may be less. So numerical preference, they tend to favor certain numbers. And my favorite one, nepotism bias. Sometimes models prefer responses from their own family of models, which is well documented. I just like the nepotism bias type stuff. There's a bunch more. What I'm trying to show here is basically how am I even approaching this hypothesis testing? So the hypothesis is LLMs could be better for, for LLMs adjust tasks. But what are we trying to improve? We're trying to improve on biases. The less biased they are, the probably better judges they are. I think it's very important to remember and to always remind you guys and myself, I keep reminding this to myself, higher evals, higher eval scores don't necessarily mean a better judge. And I've been burned by this before. So I will definitely make this case again. Higher eval scores when you see them in outputs or like in Weavle or somewhere else in JSON when you output doesn't necessarily mean a better judge. They could, but it doesn't necessarily mean a better judge. So I would love to stay on this point for just a second. If we're testing if a model works in production against your existing model, you can't just string replace, right? We would use evaluation. This is the whole course. For judges, the same thing is happening. And I love this thing. We need to use meta-evaluation. Let's go one level deeper. At some point, we'll need to stop who's going to meta-evaluate our meta-evaluation. Like at least the next level. We need to make sure that when we're testing whether or not judges are better against other judges, we run meta-evaluation. I believe that Eugene Yan maybe talk about this a little bit as well. I learned everything—

this is where the entertainment comes in, I'm waiting for it.

Yes, I've been like— everything that I know about meta-evaluation comes from directly from Eugene, so I will forward you there as well. Here's an example. This is a screenshot of W&B Weave where I got super excited and I replaced my 4.0 judge with an 0.1 judge and I saw this is doing better, right? So on, this is also a bit of entertainment. My dataset was called Doomer or Boomer where I was, this is part of my workshop as well where we're basically trying to figure out based on Blue Sky comments if somebody is a doomer, AI doomer that hates AI or somebody who's a boomer who doesn't really understand AI. So this is my dataset. I've been teaching a workshop on labeling this. And I have run a judge and I just string replaced GPT-4.0 with 0.1 and I saw a better, better score. I saw a 96% score. But then I compared it to, you know, human experts with CoinScapa and I saw that the thing did not change at all. And so basically I learned—

You had a labeled dataset of boomers and doomers?

I not only have a labeled dataset, I have made the people at my workshop actually go and label, like, live. And I built a little labeling app. I can show it later if you want. Some of the stuff that you teach, like build UIs to label stuff. So yeah, we built a Gradle thing and folks did it live. I also had a label that I said I did myself.

Amazing. Amazing.

And so this is an important point here. The Cohen's kappa didn't really move. And so basically my judge didn't improve versus the previous judge. So Meta evaluation. This is straight from Eugene Yang's LLM Evaluators and Meta Evaluation. I learned everything that I will skip this. I will basically tell you it's very important. Go learn this from Eugene or wait for him to come and talk to you about this. I'm sure that the course has meta evaluation stuff. But basically this is how we're going to judge this. And so let's actually dive into some research for LLMs and their better or worse at judging and stuff. So for use cases of math and code, Wizard models are exceptionally good at verifiable domains. They're exceptionally good at verifiable domains because they were trained with reinforcement learning on verifiable domains like code and math. So they're very good at code and math. That means that judging on those domains, so judging tasks with those domains, we can expect the judges to solve the issuing tasks at the judging process. So I'll say this again: we can expect that the judge will be able to solve the problem that kind of the model is judging, is tasked with, and then be able to also judge whether or not, you know, it was solved correctly. Um, this is, I think, very important. I've seen multiple kind of evidence to this, and also there's research to confirm this. Uh, this can address the fallacy oversight that other non-reasoners have. And users reported that first asking the judge— I'm looking at my cheat sheet, so if you guys see me looking at left, that's for some reason where my thing is. Users reported that asking the judge to first solve and then judge leads to higher accuracy as well. So these reasoning models are definitely for some of these tasks like math and code, we actually solve the thing and judge better. And because you can, and you should, afford, at least timewise, you can afford more time for your LLM as a judge. Sometimes when you have a coding agent, for example, we don't want to sit and look at cursor for a while until it does stuff. So we need latency speed. For judges, we may not necessarily need this. We just need to know whether or not our prompts are working well or our models are working well. So we can afford to give them more time to think, to solve this problem, and then say whether or not our model is actually doing this. So this on math and code, LLMs are state of the art on the judging task of math and code. I found this incredible dude, Sam Page. I'm not sure how to pronounce his last name, who created multiple things actually, EQ Bench and JudgeMark. I don't know, Hamel Husain Husain Husain, if you brought him to the course, maybe you definitely should. He built a creative writing task and he built also JudgeMark, which is a dashboard from Sam Page, incorporates a triple threat of difficult tasks that LLMs typically struggle with. Evaluating writing quality, effectively using multi-criteria numerical scoring system, which I believe the multi-criteria we don't necessarily recommend, but like it effectively uses this and following complex instructions over a long prompt. So all those three things together is becoming JudgeMark. And you can see JudgeMark are kind of online. They also have a paper and they specifically test LLM judges and LRM judges on creative writing evals. Evaluating whether creative writing is done well, multi-criteria numerical scoring. And the findings were very interesting findings. And I actually chatted with Sam a bit about this when kind of Reasoners came. I'm very interested in following up with him and seeing whether or not some of the latest models changed his thinking. But Reasoner judges can overthink and arrive at the wrong conclusion. This overthinking is something I've seen a lot, just kind of from osmosis, from talking to people, from hearing about this. But here is a person who actually did some research. And overthinking actually can make these models perform their task of judging less. And through his findings, for creative writing tests in JudgeMark, reasoners actually hurt performance. And that's very interesting. I've seen multiple folks respond with asking the model creators the ability to turn off reasoning, for example. And yeah, this is like one, one result I wanted to bring. It's not just one question about this.

So there's a multi-criteria and maybe it's potentially, it could be a complex grading task. Do you think that there's some correlation between the overthinking and the complexity of the grading or, you know, am I even understanding it correctly? Like, do you have any, do you remember anything from the paper potentially around that?

As far as I chatted with him, and also saw on EQbench and some of the other stuff that he's building around creative writing. I think for creative writing, we haven't seen these type of models excel on top of LLMs. And so I think that definitely overthinking is something that I saw. You know that feeling when maybe in Deepseeker 1 you write, hey, and you have like 5 or 6 paragraphs of it thinking before it responds to like how to reply to your word, hey, for example. So overthinking is definitely endemic with these models. But for judging specifically, I don't know, I haven't dug into like his reports. As I said, I haven't followed up with him specifically on this quite before the shoot. So I don't have like an insight of why this could be for creative writing specifically. But I know that where these models fall short is like creative writing generally and also for judging as well. But it could be due to like the multiple complex instructions and scoring systems as well. And I think that If I'm not mistaken, in Shreya's coursework, it's also recommended to do like one specific task and not multiple tasks, if I'm not mistaken.

Yeah, that's right. Yeah.

Yeah. So, so it could be that he built the kind of the judging and then, you know, gave it the very difficult tasks. But from the results, it seemed that the reasoning actually falls short on performance for this one.

Yeah. Thank you.

JudgeBench. There's another eval, there's another page, a paper called JudgeBench, a benchmark for evaluating LLM-based judges. JudgeBench released, I think, a few versions, and this one is version 1. And on JudgeBench, O1 preview— so O1 preview is the one before O1 was released back in, you know, back in December— O1 preview gets 10 points over the best non-reasoning model. 10 points over the best non-reasoning model in the overall score. This is a significant improvement on top of LLMs for judging. Now I will say that a little bit of a correlation between what I said before, JudgeBench is comprised of math and coding stuff as well, and so this could contribute to why it's significantly, significantly larger. Um, let's see what I have in my notes here. Uh, JudgeBench was created to evaluate LLM judges on tasks where crowdsourced human preference is is a poor indicator of factual and logical correctness. So judgment specifically focuses on tasks that, you know, crowdsourced human preference is not the most kind of like correct stuff. These are tasks that even hard for LLMs to do. Even GPT-4 gets a little above random 50-50 chance. So GPT-4 gets a little bit of a random chance on this. This wasn't completed yet when I tested it for O3, so I haven't seen this on like the latest and greatest. Here's also the problem with this research. I'll come back to this. Most of the research focuses on kind of the cheaper reasoning models, but we see a significant improvement even in cloud, even on top of Claude 3.5 Summit. So this was back in February, January. JudgeBench has upgraded. The cool thing about arXiv is you can go to a version of the paper and you can see sometimes the authors keep upgrading, updating the papers. With the latest models. So this is JudgeBench that I pulled yesterday. I think it's version 5, revision 5. It was revisited in April of 2025, and they've added stuff like O1 Mini, O3 Mini as well. And you can see O3 Mini is absolutely crushing this on JudgeBench. O3 Mini Medium and O3 Mini High get overall like 80% score on this and 100% encoding. I don't know how they got 100%, but yeah, the top reasoners get 10+ points over non-reasoners in this bench, even 20 points at some point. So very, very impressive performance of LLMs on these tasks that are hard for humans to do.

Was there any discussion about the 100% in the community or anywhere? That's a really interesting— it sounds like that would cause lots of discussion.

I looked at this and I haven't seen any discussion specifically on JudgeBench. I will say this is not as sexy a topic as it may seem or as I'm trying to make it out to be. In discussions, people just kind gloss. It was some amount of looking for these papers to kind of get them. I may share methodology as well because I did use deep research for a bunch of this and then kind of like used citations. I'm sure that a lot of researchers in the crowd right now look at me like, Alex, who are you? What are you doing? This is how we always do this. But this is my like learn as I go approach for you guys as well. Yeah, Hamel Husain, to your question, Coding at 100% is interesting to me, although all 3 mini at high can solve a bunch of stuff. I think they even included this in the ARC Challenge. But I think the trends is general trends and non-changing things is what we can take from this paper. Moving forward, Judgement specifically highlighted this thing. They did ablation studies, multiple ablation studies to to, to, to, to see whether or not kind of their intuitions are correct or incorrect. And this ablation study specifically showed the ability of the judge to verify the solution pairs— the solution pair, sorry— is highly correlated with its ability to solve the problem itself. This goes back to the third— the first thing that I told you. So if these models are really, really good at solving coding and math, the, the ability of the judge to verify the solution is also very highly correlated. This is a kind of a guide for you to take with you if you do want to use LRMs, if you want to activate thinking for your judging purposes, uh, the thing where it performs going to be the best is correlated to the task that this model could solve.

Um, I got a related question. Yeah, I don't know if you're going to get to this. Is there any kind of things out there about, you know, models fine-tuned with reinforcement learning for the task of judging specifically that you've seen?

I have something related to this at the end of the slides. I have not dived into the RL fine-tuning exciting area as of yet. So I definitely should probably have mentioned this, that the whole GRPO from DeepSeek R1 and other following different policies for RL specifically have improved the ability to fine-tune for Fox and make it like, you know, significantly better for judging tasks, but I haven't seen for like a specific judging task. I think broadly though, Hamel Husain Husain Husain, I think it's correct. If you, whatever, take any reasoning model and like fine-tune this with RL on your specific domain, the likelihood based on kind of this ablation study will likely rise that, you know, your judge knowing more about the field that it's judging in will likely be a better judge. But hey, that's why we have meta-evaluation for, right? Like we can run these tests and not to evaluate those judges. But I don't have this on my slides for Auriel specifically. Yeah, moving forward here, I think I'm right on time. So yeah, again, folks, I will leave you with— like, I'm not leaving yet, but like, this is the thing I want to highlight. The ability of the judge to verify the solution pairs highly correlated with the ability to solve the problem itself. I think it's very important to build when we use these LLMs to build intuition for something like this. I was very interested in this paper. This is V3 of it. I think V3 was released somewhere earlier in the beginning of the year. A survey on LLM as a judge. They did like a meta-analysis. This tells an interesting story. On their custom set of evals, they did not find O1 Mini. They did not find O1 Mini or Gemini 2 Think. Thinking to perform better. And I quote, the results presented in tables 2 and 3 yielded surprising results, insights. Reasoning-focused LLM failed to exhibit a clear advantage in tasks requiring alignment with human preferences. Failed. So we have a very like back and forth that you guys can see. And I think the reasoning— haha, pun intended— the reasoning for this is because when the evaluation tasks and judging comprised mostly from coding and maybe math stuff, these models are significantly better at this. So the survey in LLMs the Judge did not find a significant advantage in LLMs. So this was v3. And then Hamel Husain— we talked about me coming on this course maybe, I think, a month ago, and I said, okay, I'm going to show you guys the same slides. And then I did some updates, especially like last night I wanted to go and see. These folks released a revisit of this exact same paper V5, I think they released it like a month ago. And they have removed, they removed this finding recently. They removed the statement. I could not find, I went and checked this exact quote where it sits. They not only added more reasoning models there. So they tested Deepseeker-1, whereas I'm going to go back one slide. Previously they had only responses for Gemini 2.0 Thinking and O1 Mini. Maybe some of you remember, maybe not. Gemini 2.5 Pro and Flash right now are thinking models. They're hybrid models. Previously, Gemini had a separate thinking model before they kind of like merged the thinking into the main model, right? So Gemini 2.0 thinking is that separate model. O1 Mini is also a reasoning model. So they have this statement and then they tested O3 Mini in addition and Deepseek R1. And you can see Deepseek R1 still gets kind of around 56% alignment with human people here. So both these models are kind of like at the level of GPT-4 Turbo, as you can see. And this is Turbo, this is not GPT-4. Oh, Turbo is like— I think Turbo is deprecated right now. But these models don't get like significant improvements. However, they have removed this sentence. So I just wanted to highlight—

that's really curious.

Yeah. So maybe methodology was lacking. Maybe they revisited and they didn't add any improvements for these models. So they didn't highlight and say, hey, LLMs are better at this reasoning-focused— sorry, this task that we have. But they did remove this thing. So this is a lesson for me, and maybe to some of you as well. Definitely double-check if you're citing a model. Definitely double-check that you have the latest revision and the thing you're citing is still there. I was very surprised to see that it disappeared. I was like, huh. That's curious. So am I changing what I'm reporting on now? Expecting the unexpected. So for additional context, I found out about this from folks from Ryder. They built a model called Palmyra and they tested reasoners against their own fine-tune. I don't think this was RL fine-tune, but I actually don't know for sure. But it was on a very specific test set. So this is not reasoning related. Sorry, this is not judging related. I'm just giving you a general outlook of reasoners versus non-reasoners. Okay, so folks from Ryder, they trained Palmyra on financial specific task dataset and they have their own evaluation and score. And they saw that Failsafe QA, their benchmark called Failsafe QA, reasoners perform worse than fine-tuned models on context adherence than non-reasoner models. So they found that fine-tuned non-reasoners perform better than reasoners. Now, again, this is not LLM as a judge use case, but it shows that reasoning doesn't 100% translate to better at a task. And so we should keep this in mind as well.

So the message that's coming across clear to me is meta-evaluation. You can't make any definitive statements about really anything one direction or the other. You got to do some experiments and try it out. Do your meta evaluation.

Yep, absolutely. And I have more. I have more research citing— this is new. This is from April 2025, new-ish. Assessing Judging Bias in LLMs. This model specifically targets the bias point, right? So in my hypothesis, I thought that maybe because reasoning models can reason about your task, you could potentially think about maybe they present less biases. And so here's some definitive-ish replies about this. Despite advanced reasoning capabilities, LLMs are still susceptible to the biases above. They have cited the bandwagon bias, authority bias, position bias, and distraction biases. The LLMs reasoning models demonstrate better robustness than LLMs on fact-related datasets, so they can reason about those facts and they actually are more robust against fact-related biases. LRMs exhibit notable position bias, preferring options in later positions. I honestly don't have an intuition of why that is, but maybe because they are trained to see maybe the outcome. I'm not sure. But like there's, there's things here that show that they prefer options in later positions, which we know position bias is generally is a bias that exists. And they found a new one, a novel superficial reflection bias that they found and built into their thing where this this didn't exist before, but phrases mimicking reasoning within your kind of test set, like "wait and let me think," significantly influence model judgments for the worse. So if your dataset includes some kind of chain of thought stuff or things like this, this can derail the model. And so they call this the superficial reflection bias.

Do you know how exactly that superficial reflection bias, like what direction it biased? Sorry, in which direction?

I, I asked, how do you mean what direction?

Oh, like, yeah, like, did it— is it more likely the judge like agrees with the, you know, is it kind of like the, um, nepotism bias in the— in like, you know, or is it like more likely to disagree if it's Sarah's reasoning or agree or anything?

The, the thing they tested against is is kind of like whether or not it judges correctly. But I'm not entirely sure about the methodology. I didn't dive as deep, but like it affects the judge's performance, influences it for the worse, basically is the answer. And last piece here is irrelevant information derails model reasoning. They just like injected random stuff about answer A or answer B, and they were able to derail the reasoning completely away from like factual reasoning as well. These folks unfortunately tested only Deepseek. So I got to wonder whether or not this is like applicable to all LLMs or specifically Deepseek. This is generally a thing that they've highlighted in addition to these, these outcomes and these results that they've shown. They also said we, we design and evaluate 3 simple intuitive bias mitigation strategies. Specialized system prompts to reduce judging bias by almost 20%, in-context learning that provides up to 27% improvement, and then self-reflection mechanism reduces by 10%. I'm looking at this like, oh, this is incredible improvement on bias mitigation at 27% by in-context learning. And then I have scoured this paper and they did not like open source any of this work. I was very, very disappointed. So the reason why I bring you this here is because first of all, I'm very disappointed. I want to call out papers like this to say, hey, we designed and the 3 simple intuitive bias mitigation strategies, and they don't give us the strategies themselves in the paper. I really didn't like this. But also, this could give us a sense of how we can design judges that, that can improve on biases. And, you know, system prompts is one. Definitely in context learning, we should provide examples. And self-reflection is something we should definitely discuss as well. So basically, the latest thing about this, the last thing about this paper is that they only tested on DeepSeq-R1, the full one, and DeepSeq-R1 70B. And I don't love this type of research specifically because I'm sure that some of the things that they have outlined as LRMs only specific to DeepSeq. And this is also kind of a warning of mine to not over-index on some research because you should always look and see whether or not the research can include it kind of around a wide array. I've seen some other papers where, you know, Gemini thinking and ChatGPT— I can't cite the paper right now, but I've seen this like yesterday— they tend toward one place, and then Anthropic with thinking completely changes the narrative. So if a paper made decisions based on like just one model, it doesn't necessarily mean that it's applicable everywhere, but the biases still exist. All right, uh, I think this is my last paper to highlight, maybe last two. Uh, measuring self-preference in LLMs. Uh, very recent study, this came out June 3rd. You guys are definitely getting the, the exciting part. I love to report on what just happened. We're on June 10th, so this came a week ago. Uh, um, for reasoning ability, this, these people study the LLMs and find the LLMs also display self-preference bias when serving as judges. Self-preference aka nepotism, right? So LRMs also display nepotism bias. Notably, the severity of this bias is not necessarily less pronounced than that observed in LLMs. So basically, this is a whole paper showing, uh, this paper was focused on self-preference bias in general, but they tested large reasoning models and they shown that, you know, the ability to reason doesn't necessarily make the bias less pronounced than LLMs. But I wouldn't be myself if I didn't bring you research that was like, you know, last week. Speaking of last week, Apple have shaken, I don't know, my timeline, at least maybe some of your timelines as well, when they released a paper, a preprint, I don't know what to call this. It's called The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity. They introduced a Tower of Hanoi problem with multiple, multiple steps and then basically shown that models imitate reasoning while not necessarily doing reasoning at longer contexts. As tasks get harder, these models paradoxically start thinking less, not more. That's what they showed. I've seen some rebuttals on this on my timeline, so I can't go necessarily into whether or not they're correct, but definitely they've shown that, you know, some of this reasoning paradigm, the interesting reasoning paradigm, is not just scale up and the more tokens you throw on this, the better the models are. But as Hamel Husain, to your point, while this paper triggered a lot of assumptions, in our case it doesn't really matter. It doesn't matter in our case of judging whether or not the model does thinking or does illusion of thinking because meta-evaluation. We can't know either way. We just need to meta-evaluate. We have to build robust meta-evaluation processes in the pipeline. We should always compare. Now, how can we utilize the benefits of these models if such benefits do exist? So we've seen clear benefits by default in math and reasoning, verifiable domains, and multiple other properties. One paradigm that you as an evaluator must kind of consider and use to your advantage is the concept of thinking budgets in the API. Here's an example from Ader Polyglot. I think it was actually from yesterday. I think he posted this. Ader is a very popular coding agent that also does a lot of evaluations on coding abilities. Either chatbot writes its own code and also has like a lot of people trust. And actually BiggerLabs started adding AdaPolyglot coding benchmarks specifically to the releases. Right. So very, very much trusted in the field of kind of like figuring out whether models code better. The thing I wanted to bring you here is thinking budgets. So you can see at the top level, the, the top model right now that they have, uh, on this test is Gemini 2.5 Pro, the latest preview from a week ago, uh, 06/05, uh, with 32K thinking budget. What does that mean? Now in the API of Gemini, you can provide how much reasoning this model should do. So you can turn the knob of the test time compute. Basically, it's going to cost you more, but you are in control. And so when we talk about use the best model possible in the reasoning model specifically, we can talk about how much reasoning budget you provide for these models. One above this, the second model that's like, you know, closer is O3 High. O3 High, OpenAI doesn't let you provide specific like token budget, it does provide you with 3 options, high, medium, and low for reasoning effort. They call this reasoning effort. Gemini calls this thinking budget. And so Gemini also has a default thing where it decides how much thinking it needs to do. That one gets 79. But as you can see, you can force and give it like the whole budget and pay for it. You'll get a significantly better number, significantly better response. This is a great addition of LLMs on top of LLMs because As I said, meta-evaluation, we can meta-evaluate the same judge with just different reasoning tokens, right? So we can play with this as another tool in our toolset to be able to say, okay, I can make a better judge for the more complex stuff or for the less complex stuff. I may not need, I may not need as much thinking as I need, and I can run meta-evaluations against this and see whether or not it significantly improves against humans. Additional thing is reasoners-based prompting techniques. I don't have a lot of information here on the slide. I tried to really like work last night and bring you kind of specific reasoner-based prompting techniques. The thing that I found, again, it's really hard. The field is vast with folks who write very viral Twitter threads or LinkedIn posts to talk to you about like, hey, you're using this wrong. But it's really hard to find empirical data on prompting for reasoning that works like for judges. I think that area is still missing. But according to the previous paper that we saw, Bias Mitigation Techniques, we know that even for LLMs, you know, the way to improve your judges is you can do a lot of prompting techniques for judges. And so I tried to pull some specific examples. OpenAI, when they released O3, they released a prompting guide as well. Anthropic has their own prompting guides. Role assignment prompting is is great. So you can tell the judges something like, you're a seasoned university writing professor evaluating student essays. So putting the models into a role of a judge, not only like you're a judge judging this, putting them into like an actual fictional role have shown some help. Step-by-step task decomposition, structured scaffolding of your task really helps as well. Self-reflection prompting. I've added an asterisk for this. Self-reflection prompting Essentially says, hey, look at kind of like output your reasoning. Even though it's a reasoning model, it's already doing reasoning. You can ask the judge to kind of provide you with a chunk of reasoning that you would like to read yourself. And then you can ask the judge to like reflect on the reasoning that it just prompted, that it just outputted, and give you a response of why he thinks his, like their judging is correct or not according to criteria. So self-reflection. Prompting, even for reasoning models, can help. The reason why I added the asterisk here, I now remembered, is you could go one step further because some of these models actually output the thinking process to yourself. Like R1 gives you the full reasoning traces. Quen 2.5, sorry, Quen 3 gives you the full reasoning traces. You can even get summaries of traces from Gemini. You could even go one step further and do like another analysis. You can get the model to think, provide the prompts in another kind of the follow-up prompt, provide the thinking traces in the follow-up prompt and then have it reflect on its actual thinking. So that's the asterisk. I haven't tested this. This is cutting-edge stuff that I haven't tested, this last one. I just came up with it. And bias-aware instructions. You can, because these models are supposedly the reasoning models, can reason more and have a little bit better instruction following. You can provide bias-aware instructions, so you can teach them about the biases themselves and then ask them to not, to not have these biases or not to to act according to these biases. These are some reasoning-based prompting techniques that can help you. I'm sure there's more. I haven't seen a lot of research. Would love folks to kind of chime in if they have seen research. Using LLM-specific prompt optimizers is also another thing that I can help you with. I don't know if folks know about this. This is my best hidden secret. If you go to the Anthropic console for their API, you don't even have to pay them, right? Like you just go to the Anthropic, register an account. I believe for free, like I don't know if you need to pay them for this. They have a prompt optimizer. That's great. And if folks haven't used this, I really, really recommend for folks using this. This is great. I run all my prompts that I come up with through their thing just to see how they come up with this. They have this checkbox that's called this prompt will be used with models that have thinking enabled. And you can see all I wrote here is I want to prompt for Llama Judge to decide if input is great according to my criteria. That's all I wrote. And they give me a very, very nice, like, follow step-by-step judge. Now, this may not be the best LLM judge prompt out there, but it's pretty good. It's way better than the sentence and a half that I wrote. And I specifically like the— they have a differentiation between this prompt will be used for thinking model and will not be used for thinking model. Again, this is within the Anthropic kind of workshop workspace. I don't know how to call this, like the Anthropic console, basically. And you can use this one for free. I don't think you need to pay for this. This is not applying to your API keys. So basically you can run all your prompts through this thing, not necessarily LLM Judge, but definitely helps with LLM Judge as well. So the final findings, the final findings, as always, when you ask a senior software engineer about the task, the answer is always, it depends. So in our case, it depends, do meta-evaluation. But reasoners are great for topics with verifiable outcomes. So STEM topics, math and code, they're absolutely great. That's not dependable. They're better at this. They can solve the problem in their head as they think, when they judge, before they actually judge. Uh, the— according to this one study, they're not the greatest creative writing or judging. But as I showed you before, these researches can— like, the AI field moves so fast that, you know, this, this can be the case for Deepseek and some other stuff, uh, and may not be the case for O3 that's coming out, uh, like last month. Earlier models topped the JudgeBench, uh, and so Edgebench definitely shows great responses for LLMs. I've added more expensive to run because the more tokens you give, the more you pay for tokens, right? So the more thinking tokens the model outputs, the more you pay for tokens. Now, here's an asterisk that I added today because today you guys are getting a little bit of Tuesday eye like Hamel Husain. OpenAI dropped the price for O3 from $10 per million tokens to $2. OpenAI dropped the price for all three reasoning models by 80% today. So when I wrote the slides, like even yesterday, I came back and then I did this yesterday because I feel it moves very fast, right? Like it used to be more expensive to run, but like the price of intelligence is going down to zero. So you definitely should focus on the best models that you can. You should notice that these models introduce new biases, like the bias that I mentioned before that I now completely forgot its name. I'll go back and tell you. Ruling out about online eval and guardrail use. So if you remember the kind of the 3 use cases that I added, you have offline evals. Those are time independent necessarily. You can just run and wait for them. We also have online evals. So where we take a subset of production traces, for example, and then we run a certain monitor on them to make sure that we're not falling on production. Using reasoning models with high budgets, they're significantly slower, which is, Hamel Husain, to your question before, one of the reasons why Google cited to me directly that they started adding reasoning summaries in the API is because when you use those models, developers would wait for a long time with nothing. So now they're streaming the kind of the reasoning summaries. At least the developers can see something coming in. But basically the regular LLMs are streaming. The thinking models will give you the output after the thinking process. So this rules out the online evaluation and guardrail regular use. So not very easy to use these reasoning models as guardrails, because guardrails, you want the, the most immediate response possible. And the last thing that I've added is models increasingly converging. I'll tell you what that means in a second. Just before this, hot off the press— maybe not so hot because this was released maybe a few months ago— folks from Verdict, from HazeLabs, great folks that focus on LLM evaluations, released JudgeTime Scaling with Verdict with composable judges panels. And they shown in the research that based on composing multiple judges and kind of judge panels in their framework can actually get similar results as reasoning models, even higher results. On this, you can see expect-qa-judge accuracy. Verdict gets like almost 80% accuracy on this task, higher than O-1 with significantly less price, right? So they got $51 here, $51, sorry, versus O-1's $167. So state-of-the-art or near state-of-the-art results on judges. At a third of the cost. So definitely, if you are interested in composing judge panels, check out Verdict as a framework as well. So how do we build reliable LLM judges? We evaluate, evaluate, evaluate, or meta-evaluate, meta-evaluate, meta-evaluate. Hamel Husain, this is absolutely the response and the highlight of the talk. I really tried to put my face inside the volume.

That was really good.

Yeah, you should do this.

You should like do this bit, you know.

Just stand and be sweaty right here and just like, nothing about it, nothing about it. Folks, you guessed it, we meta-evaluate everything. In this case, meta-evaluation, because we always know where we stand empirically. And that's where tools come in. And Weights Biases, W&B Weave, is the tool that I'm working on. And this is the reason why my obsession with LLM evolves as well. We designed this for the entire lifecycle of tracing, agent, building evaluation datasets from production examples like I showed you before, running evals at scale, and providing a clear dashboard for you, empirical stuff. Here's a quick example. I'm sure, hoping that's not gonna be super boring. I did move my cursor like super, super quick here, but basically you can see a whole agent's execution with inputs and outputs. We're not the only ones in the field, as I'm sure you guys seen Phoenix in this course, you've seen Braintrust, you've seen other folks as well. And we, we love kind of like focusing on evaluations as well. So here's an example of our evaluation show. This is me kind of evaluating the Percy agent that we have actually running internally on kind of 3 models, the 3 model prompts that it has. And you can see kind of the same tables. Without those tables, I won't be able to tell you which, which one is the best. We can also see every sample in the evaluation and how the model performed on each sample, and also the number of trials is very interesting. So for folks who maybe given us a try before and said like, nah, I'm using this one, that one. I want to—

no, I need to give it another try because, uh, what you showed me, I'm like, wow, that's really different. It's like improved. Hell yeah. Even though there's dark mode, when do you have dark mode?

Uh, Option+M switches from light mode to dark mode. And, uh, moving forward, uh, the, the thing that I wanted to highlight is if you are like Hamel Husain and you've tried Weave before when we released this and maybe you like looked elsewhere, we did a few things since then. And I just wanted to like briefly, briefly give you a few updates and I'll promise I'll stop talking super quick. Uh, we got acquired by CoreWeave. May heard about this. Corvo is a huge cloud provider, Neo cloud provider. So we're not going anywhere and we're not, you know, anymore backed by VC. We're not chasing those gains. Basically, Weave is here to stay and we're going to work on this, like, committed to providing you the best product possible. And, you know, not a crazy rollercoaster VC ride. We're fully multimodal. So you can evaluate audio, images, and video. Video is recent. So I've seen a few evaluation frameworks for some of you who do VO3 stuff and you want to evaluate a VO3 that came came out versus v03 lite that came out like last week, which is like 10% cheaper. You want to see actually what's going on. We are fully multimodal. I found it super cool. We have MCP support. MCP is hard and like it's a hypey word, but we have MCP support both. And we will show you the MCP usage of your agent within the traces. But also we have our own MCP server and you can use our MCP server with your agent to actually see whether or not your traces ended up in Weave. And we also have kind of a support agent. If you want to implement something, you don't know how our MCP support can help you with it, this. We also have OpenTelemetry support. This is very standard across the industry. We landed this. I'm very, very happy for standard use so that if you do use Wave with OpenTelemetry, you can then— you're not vendor locked. I think OpenTelemetry is very important. Save views, I'm going to skip. And I think the coolest thing, and I think this is like the thing that I'll end like the excitement about 1B on, I've been promoting this internally for I don't know how many months. We have a new evaluation logger that's like 1B logs for evals. I think that for the people here, this is maybe the most important part about like my updates on Weavv. We basically worked to make logging evals as simple as possible into our dashboard. So you can run your evals, you can actually take any evaluation that outputs a CSV and just like iterate on top of this. We no longer require you like some other frameworks, like we used to do as well, to run our own loop. So you do whatever you do, and we will just provide you a logging thing that's very robust and simple, and we'll show you the evaluation like visualizations. So you can do your evaluation where you you want and just send them to us. This is like 1B log for evals. I really, really promoted this internally. I'm very, very happy that it landed. It makes evaluation logging very simple. I think that's it, folks. So this is—

When did that come out? I haven't seen that part before. That's really interesting. In the deep learning days, it was like, that was the magic. It was like you just add one line to your code and all of a sudden you have this logging.

Exactly.

When did this—

I won't say. We publicized about this just about two weeks ago. Ago. And I need to do more work at promoting this because it really makes logging evaluation simple. Like, literally, this is the code, right? You create evaluation logger, you log prediction, and then you log summary. And then we'll show you the same kind of visualization stuff that we have for that thing. So you don't have to do any loops, do any models. You don't have to know about our Python libraries. It's very, very, very simple. I really, really hope you guys give it a try. And please let me know if this doesn't work for you. I am advocating for you with within the folks as well. So the great convergence, will there even be reasoning models? When I started writing this talk, this was before Grok and Thinking came out. OpenAI roadmap announced O3 won't be standalone. Everything has changed. But basically what I wanted to highlight is my thesis to my talk may not even matter anymore at some point because all these models now converge. So Gemini, basically you can turn off reasoning, you can turn on reasoning. Reasoning. OpenAI's updated model, they said that the roadmap that said that 4.5 is going to be the last big non-reasoning model, and since then it will converge. So I think that's still kind of the hypothesis that I tried to come up with applies to some places, but also it applies to whether or not to turn reasoning on or off versus whether or not the model is reasoning model versus non-reasoning model. Again, when I started this, there were reasoning models separately and LLM separately. Now they're kind of You know, this feature is built in into the API. So you have to decide how to turn on reasoning on and off. This is it, folks. Thank you so much for listening. If any of this kind of interested you, evals API, this is the link for evals API. Please do give this a try and give me feedback. I would love feedback. I would love to go to my team and say, hey, I chatted with the, you know, Hamel's evals course and all these folks tried our evaluation logger APIs and they love it or they Hate it. Absolutely. It's very easy to get started with Weave. People install Weave. It's very easy. If you liked the energy or the delivery or some of the news that I gave you, the thing that I do every week for the past 2 and a half years is Thursday Night.News. You can scan this QR code and join, or join us on the live show on Thursday like Hamel often did. And that's it. Hamel, the picture of me as a judge here.

That's amazing. Thank you. All right, we can wrap it up.

Thank you, everyone.

All right, take care.

Bye-bye.