Defeating Nondeterminism in LLM Inference. Had to slop up Horace He and Thinking Machines cool work.

Ever wondered why LLM inference isn't deterministic, even at temperature 0? The common concurrency plus floating-point hypothesis isn't the full story, as kernels in an LLM's forward pass are often run-to-run deterministic. The true culprit is a lack of batch invariance. High-performance kernels change their internal reduction strategies based on batch size to maximize throughput, leading to numerically different results. Because server load is variable, the effective batch size for any single request is non-deterministic, which in turn makes the final output non-deterministic from the user's perspective. This paper proposes enforcing batch invariance by implementing kernels that use a fixed reduction strategy regardless of batch size. This is particularly challenging for attention during decoding, where they introduce a fixed split-size strategy to maintain determinism without sacrificing necessary parallelism. The authors demonstrate this approach on VLLM, achieving perfect reproducibility over 1,000 identical requests. This comes with a performance cost, but they show it's manageable. Crucially, this strict determinism enables new applications like true on-policy reinforcement learning. By ensuring bitwise identity between the sampler and the trainer, the KL divergence remains at zero, allowing for stable training without requiring off-policy corrections.