Dylan PatelFounder and Chief Analyst, SemiAnalysis · The semiconductor-AI stack

the cost of ultra-fast inference. when a single model copy requires 8 racks of wafers, a baseline de

29 May 2026 · 1 min

Watch on YouTube

1:23

This content was created by Dylan Patel. This content was uploaded by Dylan Patel. This is a YouTube video (title: "the cost of ultra-fast inference. when a single model copy requires 8 racks of wafers, a baseline de", created 2026-05-29) containing a short technical discussion between two speakers (labelled SPEAKER A and SPEAKER B). The speakers compare the capital and deployment costs of running very large language models on wafer-scale systems versus GPU racks. Topics include per-system CapEx (e.g., ~$1M per WSC), resource requirements for supporting 256 concurrent users at million-token contexts using 4-bit weight quantization and 16-bit KV cache (estimated ~24 systems / $24M, or ~8 racks of Cerebras), and a broader estimate that meaningful adoption of wafer technology could require on the order of $100M — versus buying many GV300 racks which would provide more memory and KV cache capacity for multiple model copies. The excerpt is a practical cost and infrastructure trade-off discussion for ultra-fast inference deployments.

Chapters

We've talked about how expensive these wafers are. Let's put a single WSC, like the whole system, as costing a million bucks CapEx, right? Much more expensive than a $20,000 wafer because you got to add in a whole bunch of other stuff, a whole bunch of other expenses after you just buy the wafer from TSMC or something, right? And the result is not that like you can run DeepSeek V4, this new model, on one of these really expensive systems. It's that if you use 4-bit weight quantization, And 16-bit KV cache for this model, and you want to support 256 concurrent users or concurrent requests at a time with an agentic coding workload at a full like million token context, you're going to need 24 systems, like $24 million of CapEx just to support 256 users, right? Which is like 8 racks of Cerebras in the data center just to run one copy of the model. You probably want to run multiple copies of the model. So a pretty significant upfront investment for somebody to commit to the wafer technology is probably measured in the minimum price of $100 million if you want to even start trying out how to run these really big models on these wafers.
And with $100 million, sorry. Yeah. And with $100 million, you can get so many like GV300 racks, right? And you'll have tons more memory that you can have multiple copies of the model, a lot more KB cache, right? To support huge context length.