Skip to content
Main Site News Console

AI Lab-Reported Benchmark Scores Average 6.4% Higher Than Independent Tests

· Reddit · r/LocalLLaMA Translated
Reddit

I’ve been building whatmodel.sh, an open-source AI leaderboard, and wanted to explore how the official benchmark scores published by AI labs actually compare to evaluations from independent testers.

Based on 1,629 publicly released benchmark results, I compiled and matched 611 comparison pairs across 180 models and 103 benchmarks. Officially reported scores from AI labs are, on average, 6.4% higher, and for