

Explore more content on AI as a normal technology
Treating AI as a transformative but ordinary technology, not as superintelligence.
More than 76,000 subscribers
Is AI progress slowing down?
Analyzing recent technical trends and the claims around them
Arvind Narayanan and Sayash Kapoor
December 18, 2024
169
16
29
Authors: Arvind Narayanan, Benedikt Ströbl, and Sayash Kapoor.
Since the release of GPT-4 in March 2023, the dominant narrative in tech has been that continued model scaling will lead to artificial general intelligence (AGI), followed by superintelligence. Although the most extreme predictions have since faded, until as recently as a month ago the prevailing view in the AI industry was still that scaling would continue for the foreseeable future.
Then, The Information, Reuters, and Bloomberg reported in succession that the three leading AI developers—OpenAI, Anthropic, and Google Gemini—were all running into problems with their next-generation models. Now many industry insiders, including Ilya Sutskever—perhaps the most famous champion of scaling—are saying something very different:
“The 2010s were the age of scaling. Now we’re back in the age of wonder and discovery. Everyone is looking for the next thing,” Sutskever told Reuters. “Getting the right thing to scale is more important now than it has ever been.”
The new dominant narrative appears to be that model scaling is dead, and that inference scaling, also known as test-time compute scaling, is the way forward for improving AI capabilities. The idea is to devote increasing amounts of compute when using models for tasks, for example by having them “think” before answering.
This leaves AI observers puzzled: is AI capability progress slowing down? In this article, we examine the evidence on this question and make four main points:
-
It is too early to declare model scaling dead.
-
Regardless of whether model scaling continues, the way industry leaders keep changing their tune on this issue shows how foolish it is to trust their predictions. They do not have much more information than the rest of us, and their narratives are shaped by their own interests.
-
Inference scaling is real, and there are plenty of low-hanging fruits to be picked, which could lead to rapid capability gains in the short term. But in general, capability improvements from inference scaling are likely to be both unpredictable and unevenly distributed across domains.
-
The link between capability gains and AI’s social or economic impact is extremely weak. The bottleneck on impact is the pace of product development and adoption, not AI capability itself.
Has model scaling died?
There is little new information behind this sudden shift in the narrative. We have long said in this newsletter that model scaling faces serious headwinds. Just as we cautioned against overhyping scaling then, we should now be wary of overcorrecting toward pessimism about model scaling.
“Business as usual” scaling ended after GPT-4-level models because those models had already been trained on most of the readily available data sources. We have known for a long time that new ideas would be needed to keep scaling going. So unless we have evidence that many such ideas have already been tried and failed, we cannot conclude that there is no room left for model scaling.
To take one example: training multimodal models on YouTube videos—videos themselves, not transcribed text—might unlock new capabilities. Or it might not help at all; we won’t know until someone tries, and we also don’t know whether it has already been tried. Note that this will probably have to be done by Google, since the company is unlikely to license YouTube training data to competitors.1
If model scaling is still so uncertain, why has the narrative suddenly flipped? The reason is that it has now been more than two years since GPT-4 finished training, so the claim that “the next model is just taking a bit longer than expected” has become increasingly implausible. Once one company admits something is wrong, it becomes much easier for others to admit it too. When the dam starts to crack, it can give way quickly. Finally, with OpenAI’s reasoning model o1 now out, the company has a face-saving explanation for acknowledging trouble with model scaling: they can say they are simply shifting to inference scaling.
To be clear, we have no reason to doubt the reports that many AI labs did, in fact, run much larger training jobs and ultimately did not release the resulting models. But it is less clear what should be inferred from that. Possible reasons for such delays include:
-
Technical difficulties, such as convergence failures or intricate issues in achieving fault tolerance across multiple data centers.
-
The model is not that much better than GPT-4-level models, so releasing it would feel underwhelming.
-
The model is not that much better than GPT-4-level models, so the developers have been spending a long time trying to squeeze out better performance through fine tuning.
In short, it is possible that model scaling really has hit a wall, but it is also possible that these setbacks are temporary and that eventually some company will find a way around them, whether by solving any technical issues and/or finding new data sources.
Let’s stop relying on industry insiders
It is strange enough that the new narrative emerged so quickly; it is also notable that the old one persisted for so long despite the underlying limitations…


