Dylan PatelFounder and Chief Analyst, SemiAnalysis · The semiconductor-AI stack

Sparse attention fixes memory bandwidth, not memory capacity.

16 Oct 2026 · 2 min

Watch on YouTube

1:44

In this brief technical dialogue, Speaker A and Kimbo discuss two model serving optimizations—HighCache and its sparse attention variant, HighSparse—designed to reduce operational costs for large‑scale users. HighCache aims to offload the KV cache from high‑bandwidth memory (HBM) to a lower‑tier, higher‑capacity memory, loading only the most relevant tokens during computation. HighSparse applies sparse attention to the same problem, improving memory bandwidth usage but not alleviating the memory capacity bottleneck, because all tokens must still be retained for selection during sparse attention. The conversation then introduces HiSPARSE, which combines sparse attention with caching policies that selectively load the most relevant token subsets into HBM, thereby addressing capacity concerns. The speakers emphasize that sparse attention alone does not solve KV‑cache memory capacity issues, and that HiSPARSE’s caching rules are essential for practical memory management in serving large models.

Chapters

Kimbo, do you remember when HighSparse and HighCache first got introduced? Like, these are clearly optimizations that are meant to save people money when they're serving these models, which kind of implies that the people that are introducing them have a lot of users that are using their models that they need to worry about. It's not a theoretical thing.
Yeah. So HighSparse is the sparse attention version of HighCache. And the whole idea of HighCache is trying to offload KVCache from the HBM to a lower tier higher capacity but a lower bandwidth memory and only load the most relevant set of tokens during computation. And I think HiSPARSE is a good example of an explanation to, I think, also a misnomer that sparse attention fixes or fixes all the problems of KV cache consuming too much memory or having high memory requirements. So concretely, sparse attention doesn't mitigate the memory capacity bottleneck. It's a technique that helps a lot with memory bandwidth, but as for capacity, because during the sparse attention computation, you would still need to select the tokens, so you still have to keep all the tokens in the relevant context and you cannot discard them during computation. So at some point it still has to be loaded into HBM, so the memory capacity consumption will not be significantly reduced. And that's where HiSPARS becomes useful because it can apply the typical caching rules so that you could load the most relevant parts into HBM.