Anthropic's co-founder says the biggest bottleneck to smarter AI isn't algorithms or data, but physical compute: data-center capacity and power-efficient chips. A tenfold increase in chips wouldn't give a tenfold speedup, but would still boost performance substantially.
What is the biggest bottleneck today on model intelligence improvement?
The stupid answer is data centers and power chips. Like, I think if we had 10 times as many chips and had the data centers to power them, then maybe we wouldn't go 10 times faster, but it would be a real significant speed boost.
So it's actually very much scaling laws, just more compute.
Yeah, I think that's a big one. It's like compute, algorithms, and data. Those are the 3 ingredients in the scaling laws. The efficiency with which these things run on chips also matters a lot. So we've seen in the industry like a 10x decrease in cost for a given amount of intelligence through a combination of algorithmic data and efficiency improvements. And if that continues in 3 years, we'll have 1,000x smarter models for the same price. Kind of hard to imagine.