Dylan PatelFounder and Chief Analyst, SemiAnalysis · The semiconductor-AI stack

DeepSeek V4 ships with two variants allowing it to hit 1M Context.

7 Jul 2026 · 1 min

Watch on YouTube

0:45

DeepSeek V4 reaches a 1 million-token context window through changes to how attention works. The model uses MegaMoE to speed up expert computations and ships in two sparse-attention variants: compressed sparse attention and heavily compressed attention.

Chapters

I think the headline changes or the features of V4 compared to V3 and R1 would be the 1 million context length. This ability, in order to achieve or to get to a million context length, DeepSeek did like very aggressive innovations on the attention mechanism. That's like the first thing. And the second thing is the MegaMoE to like speed up things, to speed up the expert FFN computations. First, the attention mechanism, DeepSeek-v4 comes with 2 variants of sparse attention. One is the compressed sparse attention and the heavily compressed attention.