Context Rot: Why Bigger Context Windows Aren’t the Answer for Retrieval

It’s tempting to think that bigger context windows have “solved” retrieval. Some even say RAG is dead.

But the truth is more uncomfortable: long context often makes things worse.

⚡ Recent tests (like Chroma’s Context Rot) show that when you add real-world challenges - semantic similarity, distractor facts, shuffled ordering - performance drops fast as inputs grow. The classic needle-in-a-haystack benchmark hides this, because it’s far too simple⚡

The lesson? Retrieval isn’t lookup, it’s reasoning. And reasoning collapses when the signal gets lost in noise.

The answer is not “more tokens” but better context. That means context engineering: deciding what to include, how to place it, and how to keep the model focused.

This is where Knowledge Graphs shine. With ontological structure, they can constrain the search space, disambiguate near matches, filter intelligently, and explain the choices.

Long context is a tool. Useful, yes. But the real gains come from reasoned, precise context - not from throwing the kitchen sink into the prompt.

⭕ Chroma Technical Report: https://research.trychroma.com/context-rot
⭕ Graph RAG: https://www.knowledge-graph-guys.com/blog/graphrag
⭕ Andrew Ng Bets on Graphs: https://www.linkedin.com/posts/andrewyng_build-better-rag-by-letting-a-team-of-agents-activity-7366497684976750592-ZEXU