Skip to content
Main Site News Console

AI Race: China's Open-Weight Models Trail US Closed-Source Models by Just 7 Months

· Reddit · r/LocalLLaMA Translated
Reddit

Source: whatmodel.sh, an open-source AI leaderboard I built (posted here yesterday).

Every data point comes from publicly published benchmark results: spanning 22,113 results across 987 benchmarks from 1,011 sources, with each result linking back to where it was originally published.

This chart uses data from text models: featuring 18,755 results across 617 benchmarks covering 1,211 models.

  • 14,436 results are from independent evaluations: run by paper authors themselves (primarily CC BY-licensed papers from the ACL Anthology and arXiv), Epoch AI, the LMArena dataset, EvalPlus, the EleutherAI harness, etc.

  • 7,677 are lab-reported scores published by the labs themselves for their own models.

Everything is fully open source: the dataset (licensed under CC BY-SA 4.0) and the scoring code (licensed under MIT) are hosted at https://github.com/jakemanger/whatmodel , with CSV downloads available at https://whatmodel.sh/data/manifest.json . Every number links back to its source, and the license type for each source is documented.

Interactive version (click any model to view its scores and sources): https://whatmodel.sh/race

Top Comments

sosuke (⬆69): Did you see Anthropic celebrating how they reduced distillation “attacks” https://www.anthropic.com/research/distillation-defenses ? As a user, I really hate these changes. Either way, I hope this trend continues. dennisler (⬆23): Do you actually still believe anything Anthropic puts out? It’s all just PR stunts to pump their valuation… enternoescape (⬆24): It feels like using the word “distillation” is intentionally disingenuous. I’m not saying it doesn’t happen—it definitely does—but the speed at which open-source models are catching up can hardly be explained solely by that premise. This mentality of assuming only they are smart enough to pull something off is so quintessentially American. What will they say if an open-source model stays ahead for a full month? How could that even be possible if it were just distillation? a_beautiful_rhind (⬆23): That’s not it at all. It’s really just synthetic datasets generated by frontier models, which has been happening since the LLaMA-1 days. Meanwhile, novel architectures and inference deployment optimizations get zero attention. Did they “distill” MLA (Multi-Head Latent Attention) or engrams from Anthropic? Give me a break. Macestudios32 (⬆104): Even being two years behind… owning it (even if outdated) beats renting the latest model. jakecoolguy (OP) (⬆29): I mean, models from 7 months ago were already insanely capable, so to me, this is pretty remarkable. jakecoolguy (OP) (⬆1): Agreed! jakecoolguy (OP) (⬆1): Yeah, looking at the data, I think this seems to stem from US labs going closed-source (like “Closed AI” instead of OpenAI). But frankly, what Chinese labs have pulled off here is truly incredible, and they’re only accelerating.