Dylan PatelFounder and Chief Analyst, SemiAnalysis · The semiconductor-AI stack

The inventor of Disagg Prefill, Hao Zhang, drops a new technique called Core Attention Disagg

26 Dec 2025 · 1 min

Watch on YouTube

1:25

This content was created by Dylan Patel. This content was uploaded by Dylan Patel. This is a YouTube video titled "The inventor of Disagg Prefill, Hao Zhang, drops a new technique called Core Attention Disagg" (published 2025-12-26). The excerpt is a technical presentation/summary describing Core Attention Disaggregation (CAD) and a system called DistCA for training LLMs with very long contexts. The speaker (presenter) explains the load-imbalance problem caused by attention’s quadratic cost versus other layers’ linear scaling, and presents CAD’s approach of isolating stateless core attention, partitioning attention into fine-grained token-level tasks, and dispatching them to shared attention servers (idle GPUs) with a ping-pong execution scheme to hide communication latency. The talk also notes pipeline-parallel integration, repurposing GPUs during pipeline bubbles, and reported throughput improvements up to ~1.35x on workloads up to 512K context length, with larger gains for longer documents and larger scale.

Chapters

Training LLMs on long contexts creates severe load imbalances, leaving expensive GPUs idle. This happens because core attention compute grows quadratically with sequence length, while other layers scale linearly, creating stragglers. This paper's solution is Core Attention Disaggregation, or CAD, which isolates the stateless, parameter-free core attention computation. The attention workload is partitioned into fine-grained token-level tasks and dispatched to a shared pool of attention servers. These servers, which are just other GPUs temporarily taking on this role, can re-batch tasks from any data or pipeline parallel group to achieve near-perfect load balance. But what about the communication overhead from this dispatch? The system, DistCA, uses a ping-pong execution scheme to almost completely hide this latency behind the computation of other layers. KAD also integrates elegantly with pipeline parallelism. It repurposes idle GPUs during pipeline bubbles to process attention tasks, increasing overall utilization. On workloads with up to 512K context length, DistCA improves training throughput by up to 1.35x over strong baselines. The performance advantage generally increases with longer documents and at larger scales. By disaggregating the core computational bottleneck, this method offers a more balanced path for scaling long-context training.