Dylan Patel breaks down the economics of LLM inference accelerators. Tuning GPUs for speed gets you 2-2.5x faster at 6x the cost, while SRAM-based machines like Cerebras could hit 10x speed but charge 20-50x more per token. The question is whether anyone will actually pay that premium.
Chapters
Yeah, people are willing to pay more for speed just with tuning for interactivity on existing HBM-based accelerators like GPUs and TPUs, where the trade is like 2 to 2.5 times faster for 6x the price. But Cerebras kind of extends that to like potentially 10x faster, maybe more, depending on the model and depending on the profitability you want to run the endpoint at, and potentially exponentially more. I mean, we don't know exactly what the profitability of the 4 5.3 Codex Spark endpoint with OpenAI is, that's the biggest public example of people using Cerebras for inference. But if you say conceptually, what are the SRAM machines promising? It's generally 10 times faster at some exponential on top of 10 times the price, maybe 20 times the price or up to like 50 times the price per token. It'll be interesting to see them play with that part of the Pareto curve as this gets rolled out.