I’ve been building whatmodel.sh, an open-source AI leaderboard, and wanted to explore how the official benchmark scores published by AI labs actually compare to evaluations from independent testers.
Based on 1,629 publicly released benchmark results, I compiled and matched 611 comparison pairs across 180 models and 103 benchmarks. Officially reported scores from AI labs are, on average, 6.4% higher, and for