Can LLMs Judge Role Play? PersonaEval Tests Them

More and more papers are showing usage of LLMs as role-play characters, some for entertainment, market analysis, or even behavior world simulation. All of those use cases require being able to evaluate the quality of such role-play simulation. We could rely on human, but as we know, humans are slow, expensive, and not scalable. As a result, we often end up using LLM as a judge, as evaluation process. But this raises a question: are LMS judge reliable? Obviously, there are many papers who are exploring these questions, but none are targeting this idea of role-play simulation. Until lately, August 2025, with the release of Personal Eval. Personal Eval is a very interesting paper that I came across totally randomly and based itself in a CIPOL hypothesis. An LLM cannot credibly assess how well a role is portrayed if it cannot first identify who's speaking. In practice, what it means is to be able to assess if a simulation is correct, one must first be able to know who's we're trying to imitate in this simulation. And as simple as it can seem, no one has taken the time to validate this hypothesis. And that's exactly what PersonaEval is about. PersonaEval is a benchmark validating that LLMs can detect who's the person currently speaking in this simulation. But how? In practice, it's pretty simple. We'll show an LLM a two-turn conversation. The first turn from a known character, and second turn from a second unknown character that the LLM needs to guess who it is. Then we'll present 4 options of who this target character can be, and then we have to select who it thinks the current second term character is. We do that hundreds of times, and at the end we come with the accuracy. But how to build such a pipeline across hundreds of turns? So first step is to come with raw data. It can be from novels, from scripts, or even from videos. We retrieve all the communication turns of this raw data and we'll first filter out all small turns, meaning every turn that is below 25 tokens. This ensures we remove every non-meaningful conversation like, "Yeah, sure, amazing," that doesn't provide any information into who's the current speaker. Then we want all 4 candidates to be challenging for the LM. We don't want one to be very obvious while the others are non-relevant to the current context. So what we will do in practice is we'll find a target speaker, we will embed it as well as all other candidates, and we'll do cosine similarity to find the most closest candidates compared to our target speaker, ensuring that all 4 candidates are meaningfully challenging for the LLM. Finally, we build a prompt where we present the first turn of communication. We squeeze the character currently speaking, a bit of context about the situation, and abuse his transcript. Then we provide the transcript of the second turn without saying who is speaking. And finally, we present all four characters asking the LM to pick one, as well as confidence assigned to each candidate. However, when doing this pipeline, authors realize that in many cases, the answers was too abused. Either because the answer was directly included in the transcript or because the context was too much leading. In order to provide a meaningful benchmark, they wanted to filter out all those samples. In practice, they ran DeepSig on the whole benchmark and filtered out every sample that led to a confidence accuracy over 50% on the correct candidate. This resulted into 3 tracks. Literally, based into English novels. Drama, based on Chinese screenplays. And finally, Expertise, based on the TV Wired Five Levels videos. So what are the results? Are LLMs far beyond human expertise level? Are LLMs way below humans' levels currently? It seems like an easy task, but it requires some background into human communication and thinking process in general. Here are the results. On average, the best LLM performed at 68.8% accuracy. It was Gemini 2.5 Pro, a multi-billion parameter model. And more generally, reasoning models performed way better than their non-reasoning model counterparts. But before comparing that to human results, let's take a look at a few experiments I've tried. First one was training time adaptation. In practice, I built a dataset composed of 5 tasks. World knowledge, identity recognition, dialogue assignment, behavioral alignment, and instruction prompting. And I wanted to know if fine-tuning in such tasks could help the LM to perform on Personal Eval. As a result of following, in all cases, it only degraded the results, meaning that training-time adaptation doesn't help in Personal Eval. The other way they explored is test-time compute. With few-shot prompting and self-consistency. And here we have a bit different results. Few-shot prompting meaningfully improves the results where self-consistency do not meaningfully impact the results. This seems very interesting and promising for LLM results. However, few-trunk prompting is dependent on the quality of the selected example as well as their ability to generalize. But now, time to human result. How are we performing in such a benchmark? To answer this question, Seth handpicked 10 undergrads and 10 PhD students. Seth handpicked the 50 most difficult examples, and Seth asked all those 20 candidates to answer those 50 examples. Here are the results. Humans perform at 90.8% on average. Far exceeding any top-performing LLMs. So what conclusion should we have from all that? Are LLMs bad as a judge? Are LLMs bad as role-playing? Well, the answer is more nuanced and more gray than that. For now, this field of research is really promising. However, it's important to keep in mind that we're still in the early days of AI and we still have a long way to go before being able to meaningfully simulate human behaviors. If you like this presentation and want to explore other papers, feel free to subscribe where we'll explore other papers.