Angelina YangFounder and Host, TwoSetAI · AI builder for visibility and discoverability

Why 10 Million Tokens Won't Save AI (The Attention Problem)

19 Dec 2025 · 1 min

Watch on YouTube

0:30

Short excerpt from a talk arguing that merely increasing transformer context windows (to millions of tokens) won't solve fundamental issues. The speaker identifies an "attention problem": transformers cannot have unlimited context while retaining precise understanding, so the priority should be providing AI with the right, not just larger, context.

Chapters

Models are piling up their context windows, right? There are eventually going to be models with tens of millions of tokens available for the context window. But the problem is, with transformer structure, it's impossible to have unlimited context versus very accurate understanding of context. There's the attention problem. And so we don't believe that maximizing the context window is going to be the ultimate solution. We believe that giving AI the right context is the step one.