Context Rot: When Long Context Fails

Welcome everybody. Today we have Kelly Hong, who is a researcher at Chroma. One of the things that she has published recently is this article that she has shared on the screen, Context Rot. This research took off and you may have already heard this term. People have started using the word context rot, but I think it's pretty useful to hear where it came from. And also this research is really interesting because it has some pretty surprising results in it, which we'll dig into. So with that, I'll hand it off to Kelly.

Thank you so much for having me here today. Uh, yeah, I'm Kelly. I do research at Chroma. So Chroma is a vector database company, but we also do some applied AI research, such as our Context Rot report, which we published back in July. But basically through this report, we demonstrated that models have non-uniform performance across input lengths. So even if you're working with a model, say you have a 1 million token context window, it's not guaranteed that you'll have reliable performance with maybe a 10 10,000-token prompt versus a 1 million-token input. And this is not necessarily a new phenomenon. I think this is something that a lot of people have experienced personally in their daily use. For example, maybe if you're working with a long-running coding session in Cursor, or you have a long conversation in ChatGPT, you might start to notice that at a certain point, you start to get lower quality outputs and your model doesn't perform as well as it did earlier on. So that was our main motivation behind this. We kind of wanted to quantify this degradation in performance beyond just like qualitative observations and also to motivate people to think about their models and use them in a more thoughtful way. Yeah, I'll start off by going over the current state of long context models. So in a lot of these frontier model releases, like you see Gemini 2.5 Pro, Claude Sonnet 4, GPT-4.1, they all highlight their 1 million token context window. Llama 4 even has 10 million tokens, which is a lot. But the way that these context windows are typically presented is that they imply that these models maintain performance across all input lengths. And if we assume this to be true, then obviously it makes sense that you'd want to give your model as much context as you have, use up as much of that 1 million token context window, because if it's able to use all of that information efficiently, then obviously your outputs are going to be more informed and more detailed. But, um, with the Anthropic release, they talked about how if you have this 1 million token context window, you can do the same coding tasks, but with larger codebases, you can maybe input like hundreds of legal documents, get more detailed results from that. And this all sounds really great. It doesn't seem to have any downsides if you have the assumption that your models are gonna maintain uniform performance. And a common benchmark that is used for long context is this test called Niro Nihistat, and I'll go more into what this task actually is in the next slide. But just by looking at a visualization like this, you can just immediately understand that from 100K input tokens to 1 million input tokens, you have success across all input lengths. So all this kind of carries the implication that We're giving you this huge context window, you can use it however you want, and we're gonna give you high performance regardless of how long your input is. But I wanna talk a bit more about what the Needle in a Haystack test is. So it's a very simple retrieval task. You basically place a random fact, which you call the needle, and you place that in a long context window, which we call the haystack. And then you just ask the model to retrieve that fact by asking a question. And we use this example here where we have a haystack that's constructed from Paul Graham essays. And then you have this question-needle pair. The question is, what was the best writing advice I got from my college classmate? And the answer to that, the needle, is the best writing advice I got from my college classmate was to write every week. But if you look at this needle-question pair, you can see that there's a lot of lexical overlap. You don't really even have to understand the meaning or like semantics behind the question to perform this matching because it's just direct lexical matching. But the reality is that you're gonna face tasks that are much more complicated than just simple lexical retrieval, and they're gonna be more semantically oriented as well. Semantically oriented basically means that you're not gonna have these perfect lexical matches because you probably don't know ahead of time exactly what you're looking for. You want the model to infer these semantic associations. And that was the first test that we did. We added ambiguity to the needle question pairs. So if we take the same question, what was the best writing advice I got from my college classmate, a lexical match matching a needle would be, "The best writing advice I got from my college classmate was to write every week." But if you have a more semantically matching needle, it might be like, "I had an interesting friend who I took some humanities courses with back in college who would write every single day, and he told me to try writing at least once a week. Looking back, I think it's the most useful habit I've developed for my writing." So if you have this kind of needle in your haystack, you'd have to make the association that the college classmate connects to the friend who I took some humanities courses with back in college. And the best writing advice connects to the most useful habit I've developed for my writing. So you can't just perform like simple lexical notches on this. And with this, we've shown across the models that when we have these like semantically matching needle-question pairs, our models are a lot more unreliable, especially as you scale in input length compared to if you have lexical notches. And I think something interesting to point out here is that at the start of this, so when you have like shorter input lengths, the models perform pretty similarly regardless of whether you have lexical matches or semantic matches. But it's like only as you scale in input length that you're able to see this divergence even more. And—

Explain this, this visualization. Like what's the difference between the 4 models and the 9 models? Are these the average across 4 and 9 models?

Yes. So it's average across 4 models here, and we have these 4 separate graphs because the models have a lot of variance just based on like their performance. Like for example, GPT like 4.1 is a lot like higher quality than say GPT like 3.5 Turbo. So that'll be considered like a lower performing model. So then yeah, we basically like filter this by like the model's overall performance and it's like context window as well. Yeah, because some models they have a 200K token context window, so we just filter for that. But Yeah, just averaging across that group.

And I have a quick question.

Yes.

You know, for people that have not looked into the benchmark closely, they might be really surprised that it's like a lot of lexical matching. Does that mean that benchmark was pointless all along, almost?

I think that benchmark, initially it was pretty useful because when it tested with the older models, maybe from a year ago, we actually did see this like performance degradation. But as these models got a bit more advanced, like they were able to perform perform very well, as we can see here. I think Needle in a Haystack is a good test because it's like very scalable, but like we're able to make these like slight modifications and show that, oh, it's actually like not, maybe not like a representative task. So I think like the naive like Needle in a Haystack task is not representative of a real use case. So you shouldn't just look at this and think that, okay, it performs well on Needle in a Haystack, that means it's going to perform well on any other complicated task. So I think it's just like important to keep that in mind. But I think this task, like, allowed us to, like, create this variation. So I think, yeah, I would like to, like, credit the work for that. But yeah, just important to keep in mind.

Makes sense.

Yeah. And I want to point out the implications from this as well, because if you think about real applications of when you use these LLMs, you're often going to use more ambiguous queries than perfect lexical matches. And I think a good example of this is if you think about a financial report analysis. Maybe you put in a few, like, 100-page documents into your LLM, and then you ask it a very specific question that refers to this snippet in the document. It says, Germany contributed $45 million, Japan contributed $35 million, United Kingdom contributed $32 million, combined international markets totaled 67% of revenue in fiscal 2024, up from 43% in fiscal 2023. And If you have a lexical matching query, you might have something like, how much did Germany, Japan, and the United Kingdom contribute to revenue in fiscal 2024? You can see that like Germany, Japan, and the UK, they have, they can like directly match from the document snippet and also like revenue in fiscal 2024 here. But it's very unlikely that you're going to know all of this information beforehand. If you're working with 100-page financial report documents, you're not going to know all the details that exist in there. So you're more likely to ask a question like, how is our overseas expansion going? Just something a bit more ambiguous. and then you want the model to extract this information for you. So just something important to keep in mind that, you know, when you have this like greater like ambiguity, your, your models are going to be more unreliable as you have longer inputs. And we also did another test where we added distractors, and I first want to define what a distractor is. So if we take the same example of the needle in a haystack, We have a question that's asking about writing advice, and we can break the haystack down into two types of content. Majority of haystack will be irrelevant content. So this haystack is made up of an essay about addiction from Paul Graham, and this is obviously like completely unrelated to the needle that we have about writing advice. But a distractor is something that's semantically similar to the needle. So it might say something like, I think the best writing tip I received from my college professor was to write every day. It's a fact that sounds similar to the needle and might be a potential answer to the question, but it doesn't fully answer the question, which is why we consider it to be a distractor. And so we have this needle-question pair, and these are the 4 distractors that we use for this variation. This one just replaces college classmate with college professor. This one talks about the worst writing advice. This one's about classmate from high school, and so on. So we do this experiment across 3 different conditions. The first is where we just have a needle with no distractors. The next is where we have only 1 distractor in the context, and then the last condition is where we have 4 distractors along with a needle. And so we see through this visualization here that as you have more distractors, your models become more unreliable. And this, um, these models become more unreliable as you have longer inputs as well. So the orange line represents the 4-distractor condition where we have all 4 of these distractors in context. The yellow line is where you have 1 distractor, and the blue is where you just have no distractors, only the needle. So you can see that if you have very similar information in your context, the model tends to get more uncertain. It might hallucinate sometimes, or just say that it doesn't know what the answer is, which leads to this degradation in performance. And we see that as you scale and input length, the models have even more uncertainty. We— I'll talk a bit more about the failure cases later, but I think the important takeaway from this is that it's very common that you're gonna have similar information in your context, especially if you're working in domain-specific applications. For example, if we take the financial analysis example from earlier, let's say you have a question like, okay, how much did Germany contribute to revenue in fiscal 2024? The answer might be Germany contributed $45 million in fiscal 2024. But you might also have some distractors in the context. You might have a document that's mentioning how much Germany contributed back in 2023. You might have a fact about France as well. And I think this financial document case is especially interesting because for a lot of financial reports, you're going to have like very standardized formats. Maybe like the only differences is like the number or the names that are mentioned. And this is gonna make it more difficult for the LLM to kind of distinguish between the similar information. And we show here that when you have these distractors, like even if you just have 4 pieces of similar information, that degrades performance pretty significantly for the frontier models today as well. And we did some more investigation into the failure cases of these models. And so as I mentioned before, if the model doesn't answer correctly, it might either say, okay, I just don't know the answer, so it abstains from answering, or it hallucinates and answers with a distractor instead. And then we basically like compare the different types of responses from each model family, and we see some pretty interesting behavior patterns. Like for example, in the Claude model family, we see that the model tends to abstain more from answering whenever it's uncertain. So it'll just say, I don't know what the answer is if it's not confident of its answer. And we see with GPT models that they're actually, they have the highest rates of hallucinations. So when it's like uncertain about an answer, rather than saying that it doesn't know the answer, it'll just answer with an incorrect answer. And this is pretty important to keep in mind because as you have like on-ground tests, we show that your models are more prone to hallucinating wrong answers, maybe if you're working with like GPT models. Just something to be aware of because I think people often just assume that if you give all the right information in the context, you want the model to perform well, you expect it to perform well, but we show that it doesn't even on this very simple task. And another experiment we did was with the kind of like shuffling the content of the haystack itself. And this experiment specifically isn't directly applicable to a real use case, but I think it shows an important point that you can't really make assumptions about these LLMs because they might not behave in the way that you always expect it to. And y'all, let me explain some of this in more detail. But basically we have the original task where you have the Paul Graham essay. So this one is the essay about addiction. And in the haystack, you can see that the sentences have like a logical flow to them. You know, it's, and as the sentence in this, but then like suddenly it just talks about like random, like writing advice they got from their college classmates. And so with respect to, I guess, the structure of the haystack, this needle stands out more. But then we tested in another case where we have a shuffled haystack. So this haystack contains randomly shuffled sentences from various essays. So we might start off with something about addiction and then starts talking about founders and then philosophy, and then you have a needle about the writing advice. So if you just think about, I guess, like the logical structure of this haystack, the needle seems like it would blend in more with a shuffled haystack. So you would think that, okay, maybe like the model would have a harder time picking out this needle in this shuffled haystack. But what we saw with this experiment was that the models actually perform better when the haystack is randomly shuffled, which seems almost counterintuitive because you might think that if these models process context in a more structured, order-sensitive manner, it would be able to pick out this needle like a lot easier when it breaks the logical flow of the surrounding context, but it performs better in this randomly shuffled context, which kind of goes against maybe assumptions that people had previously about these models.

I have a question, if you don't mind, about this. What about like realistic needles, let's say, that do flow in the, that are logically related to the context around it? Is there any research on that?

That's a good question. I, yeah, we were thinking about, we wanted to experiment with that as well. The difficulty with that is it's hard to keep the difficulty of the task like consistent across like needle position. So maybe you have, you start off with an essay about addiction and then you have an essay about like founders. You would need to write an essay for addiction and then write an essay about like founders. But we can't guarantee that like the difficulty of matching like the needle to the question is going to be similar for every single topic that you have, which is why we didn't test that. I think that would be a very interesting test actually, because I'll like test whether, okay, if the needle like actually like blends in with the essay that is embedded into, like how would that make the task more difficult? Like how would it compare to the shuffled context? So yeah, I think that would be something interesting to test, but just difficult to control, especially for like longer text. Makes sense. But yeah, I think, yeah, from this, as I mentioned, it doesn't really have a direct like mapping to a real use case, but I think it goes to the point that you can't make certain assumptions about these models, like even if it logically makes sense. So I think it's important to kind of take a step back and realize that these models aren't always going to behave in the way that you want it to. And yeah, unfortunately we don't really know like why this happens. As with a lot of LLM behaviors, I think this would require a lot more research into interpretability, but just, I think, one of the more interesting findings that we had from this paper. And we did a few more experiments as well. I won't go into too much detail, but we have the full experiments and results in our research report if you want to read it later. But another test we did was on conversational question answering. So this is a bit representative of a case where you maybe have a long-running conversation in ChatGPT, and it's essentially like the same task across like both conditions. Like you just have some like relevant like chat conversations and then you end with a question that like asks about some earlier chats. So it's just like similar to like needle in haystack, but more like conversational. And we test this across the focus condition, which is the red bar here where we give the LLM only the relevant context. So these inputs usually range like around 100 tokens, so very like small and focused. And then we compare that with the full condition where we give the LLM inputs of around 120K tokens. So we have the relevant parts, but then we fill the rest of the context with just unrelated chats about like other random topics. So the only difference is like context length, and we don't add in like any distractors. It's literally just like irrelevant content, but we show that Um, but even with the latest like Claude, um, models, they have a pretty big distinction between the full and focused context showing that even if you have the task of a similar level of difficulty, its performance is gonna matter based on how long of an input you give it. And then we also did another test on text replication. This task is basically just to replicate a piece of text that we give to the LLM. So the input is literally just simply replicate the following text. Output the exact same text. And then we give it a series of repeated words with a slightly modified word in the middle. And this task was especially difficult for the LLMs. We see that, like, even with the frontier models, they degrade in performance quite significantly. And we saw a lot of interesting behavior from this as well. Like, sometimes with, like, long inputs, the Claude models would think that it's, like, generating copyrighted material, so it would just refuse to generate any response altogether. And then with the Gemini models, we started seeing just completely random outputs, I think, once we got to around 5,000 input tokens, which honestly isn't that much if you think about it as 1 million token context window. And I think just taking this all together, even if we have relatively simple tasks, like just simple fact extraction or text replication, you would expect these models to do well even on long context length, but we show that they have non-uniform performance. And you can imagine that if you have a more complicated task, if you have more ambiguous queries, maybe these models will perform just as worse or like even more worse than they did here. And I think the important point I want to make here is that even if you give your LLM like the right information and context, like how you present that matters a lot. So this includes like whether you have irrelevant information in your context, if you have distractors, So you need to think about like how you engineer the context before you give it into your model. And I think context engineering, it's highly dependent on use case. So there's no like general like context engineering solution, but I'll just walk through one example that I've seen being used in practice just to give a sense, but it's really dependent on your applications. Yeah, definitely think about that specifically. But for this use case, say you have a coding agent and you have a long-running task. A naive approach to building this kind of agent would be to continuously append the conversation history with every single tool call and turn you have, and you just keep everything in your context. So this would include like a tool call, like a file read, for example, which has a pretty long output, and that's gonna make your context grow very quickly to like thousands of, hundreds of thousands of tokens. But a better approach to this might be to maybe break the task down into subtasks and then use subagents. So you can just think of each like subagent involved in this as like an LLM with its own like conversation thread. So you start off with the main orchestrator agent, which breaks down a complicated task into a plan of subtasks, and then it spawns a subagent to complete each subtask. And then because each subagent has its own context, you know, it's able to perform that very focused task and then only return the most relevant information back to the orchestrator agent. And then this kind of allows the orchestrator agent to keep a clean filtered thread of the most relevant information. So this is something that I've seen being used for coding agents or maybe even like deep research agents too. But there's a lot of different approaches to context engineering. I think there's a lot of talk around that now too, but yeah, I think that pretty much covers ContextDraw. We have more experiments and details that I didn't fully go over in this presentation, but you can view the full report at research.trichroma.com. We have some other reports available there as well. We have like generative benchmarking, which I actually talked about before on this course as well, but yeah, thank you so much.

Yeah, I would love to ask you some questions, if you don't mind.

Of course. Yeah.

Is there any—

do you—

I think you have in your report a breakdown of models and how each one degrades differently across different lengths. Do you mind pulling that up? Or if you don't mind.

Yes. Nice. We do that with the repeated words experiment. We do that with like Longman eval, the conversation eval.

Isn't the one at the very beginning comparing models at the—

Yes.

Very beginning.

This one?

Yeah, I think so. Okay, that's the repeated words.

Yes, it's the repeated words experiment.

Did you find that— I think one burning question is, did you find that one model is like dominating the ability to resist context rot, like across the board on these? Because you did a battery of different tests and we can look through this report and dig around, but did you find, okay, this one model seems pretty, pretty like good relative to the others, or was it like all over the place?

It was all over the place. There wasn't any model that ranked first and like across all the tasks. I think for the repeat-a-word tasks, like Claude-Saundre Ford did the best. But for a task like needle in a haystack, GPT-4.1 did the best on that. So I think each model has its own advantages and I think it's highly dependent on the use case. Unfortunately, there is not a model that is like excelling at long context currently.

And so one thing that sticks out to me here is is when I think when people hear the term context rot, especially the word rot, I think, okay, like the word rot is associated with this idea that you have to like really neglect something before it starts rotting. But what you've shown here is it's actually not that hard to perturb the model, or it degrade— can degrade like quite fast on these various tasks. And it seems to me Okay, even though a lot of these models have million token context window, it's quite easy for them to degrade and it's not necessarily about neglect. It's, hey, you've stuffed too much, just too many things into the context. And I think there's this series, there's a, this RAG series that we're doing, people declaring like RAG is dead. In response to, hey, you can put all this stuff in context, maybe you don't need RAG. So what are you seeing out there? Are you seeing that people are backing off that? Like they're taking this context and being more parsimonious about what they're putting in context? Like they, do you see like a swing towards putting a lot of stuff in context and then not working? And then now we're back to, okay, like doing proper retrieval again, or what do you see in practice at Chroma?

Yeah, I think, I think a lot of people say that RAG is dead because when they think about RAG, they only think about like vector retrieval. You have some sort of like vector database, you just do like semantic search and then you pass that into your LLM. But I think like, I think of RAG more like generally, like maybe you can even just use like an LLM to do filtering for your context before you like pass it in for the actual task. So I think maybe a better term for it is like context engineering that's becoming like more popular than like RAG or like prompt engineering. So I think that is a trend that I've been seeing a lot, like internally, like we talk a lot about like context engineering here at Chroma. I think just like online as well, if people are just talking about optimally like using LLMs, they talk about like context engineering. You want to give it like the most relevant information. The ways in which you do that is not only vector retrieval, there is So many different methods, like the sub-Asian approach I talked about. So I think like people are coming to the consensus that like context engineering is important. So I think maybe like the terms are changing a bit to shift away from like solely vector retrieval.

What would be your advice to people who have AI applications that, you know, have RAG pipelines, which is basically like everybody working on RAG app or AI applications? How should you be paying attention to context rot? How should you try to find if it's happening? What do you look for, etc.?

I think it's like pretty easy to spot. I think even just like running like a few examples, like long context, like maybe you give an example with a few like 100K tokens, but you give the same task with like, like a few like 100 tokens, like you just fill the context with like irrelevant information. You can just look at like the outputs of your model and then see that it's like performing worse. I think it's ideal to have a more like robust evaluation system, but I think to start off, just start by like looking at your inputs, like how long your inputs are, what the outputs are of that, like what did the model miss, what kind of like information was like irrelevant, what can you like take out from that. I think just starting off by looking at like the kind of like data that you're working with. And I think that's like the, that would be like my recommendation for like starting point because I think like how you do like context engineering is like so dependent on like what application you're building. So I can't give like one generalizable answer, but I think you can get a lot out of just looking at what you're like inputting to the model, the outputs you're getting out, and like how it could have done better.

Sounds good. Okay, I'll ask you one question from the audience.

Yes.

So there was some prior research, if you recall, about long context and how content at the beginning of the context and also at the end of the context was more likely to be found than if it was like at the middle. Do you know anything about if that's still true? Is that U-shaped curve like still hold or do you know anything about that?

That's a good question. I know there has been a lot of prior work on that, but through our experiments, we actually didn't see any advantage to putting important information in the beginning or the end. Like, especially with, like, our needle in a haystack experiments, we did that across, like, different, like, percentages. So we might have some in the beginning, and then we just did that across, like, the entire haystack. So we tested, like, basically almost every, like, possible position, but we didn't see really see any pattern in what the model favors. And then for the repeated words task as well, like this task here, the reason why we have the modified word was to see whether the model is able to detect this maybe more frequently if it appears in the beginning or the end. But actually, with this too, we also didn't really see a huge trend in whether the models favor the beginning or end. I think people like give that advice because it's like easy if you're working with like long context, if you can just put the important information in the beginning and then just give it like all like possible like relevant context and then just like hope that it works. But unfortunately, like that doesn't solve all your problems and you have to be a bit more intentional about how you manage that.

All right. Sounds good. This is very interesting. Kelly.

Yeah.

Thanks for doing this. I'll be publishing notes. On this as well.

Okay.

And so thank you so much.

Thank you so much for having me here today.

Thank you.

Thank you.