Dylan Patel breaks down SemiAnalysis' InferenceX V2 benchmark comparing AMD MI355X and NVIDIA B200 on large language model inference. AMD's hardware matches NVIDIA on raw performance-per-dollar, but NVIDIA's software stack and NVLink's 72-GPU bandwidth create a practical advantage that matters in production systems.
Chapters
SemiAnalysis dropped InferenceX V2, the most comprehensive open-source LLM inference benchmark ever built, testing every major NVIDIA and AMD chip from the last 4 years. And the results tell a very clear story about where the real moat in AI actually is. InferenceX benchmarks frontier mixture-of-experts models like DeepSeek R1 at scale with real production techniques like disaggregated prefill, wide expert parallelism, And FP4 quantization. AMD's MI355X is genuinely competitive with NVIDIA's B200 on FP8 disaggregated inference, and in some scenarios, it actually delivers better performance per dollar. The hardware is there. But when you combine FP4 with disaggregated serving and wide expert parallelism, which is what Frontier Labs actually run in production, AMD falls behind because of software composability issues. That That is, each optimization works alone, but they break down together. Nvidia's moat isn't the silicon; it's the software. The GB300 NVL72 provides a generational leap, delivering up to 100x better performance per GPU than the H100. Jensen promised 30x at GTC 2024. Turns out, he underpromised and overdelivered. The secret? Rack-scale NVLink. 72 GPUs? GPUs connected at 900 gigabytes per second, over 9x the bandwidth of InfiniBand. That lets you run wide expert parallelism without ever leaving the NVLink domain, loading the 670 billion parameter DeepSeek model once across all chips instead of replicating it. Rack-scale Blackwell is frame-mogging Hopper so hard they needed a log scale on the dashboard to even show both on the same chart. Bottom line? AMD has the chips to compete. Software is the moat, and Blackwell at RackScale is in a league of its own.
