Skip to content
Main Site News Console

Can Frontier Models Spot the Decoy? Elorian Tests Visual Reasoning

· Latent Space Translated
播客深度访谈

A new decoy counting benchmark exposes the disconnect between how machines “see” and how they “reason.” With Glimpse, Elorian attempts to bridge this gap by “thinking with images.”

Richard MacManus

Oct 29 · 6 min read

Image 4

This month, Elorian introduced its first visual reasoning model, Glimpse. It is described as “a new class of visual thinking model: systems that reason about the physical world based on what they see.”

View post on X ↗

So, how does Glimpse stack up against frontier models and world models in vision benchmarks? DecoyCount, a new benchmark co-released by Elorian, offers some answers.

We spoke with Elorian CEO Andrew Dai about the benchmark and discussed how Elorian positions itself in an increasingly competitive model landscape.

DecoyCount Results

Elorian emerged from stealth this April with a $55 million seed round co-led by Striker Venture Partners, Menlo Ventures, and Altimeter Capital. Now, partnering with AI data firm Mercor, Elorian has released DecoyCount—a benchmark designed to test whether language models are truly counting objects or merely relying on visual shortcuts and proxy heuristics.

According to an accompanying paper released alongside the benchmark, frontier models all scored below 50% accuracy on DecoyCount—with the top performer, GPT-6 Astra, hitting 43.17%. While Elorian’s new model was not mentioned in the paper, Dai told us that Glimpse achieved 50.48% in internal testing.

Regarding the frontier models, the paper concludes: “Stronger general reasoning capabilities do not inherently generalize to aligned visual reasoning.”

Image 5

How the benchmark works: “DecoyCount reveals interpretable counting failures across seven mechanisms.”

To ensure an apples-to-apples comparison with the frontier models, we asked whether the evaluation images in the paper were held out from Glimpse’s training data.

“Both the evaluation images and text were excluded from our training, and our contamination checks revealed zero overlap across the inspected training data,” Dai responded via email. “Without any prompting cues, Glimpse scored 50.48% across the 1,660 DecoyCount items, making it the only model in this comparison to break the 50% threshold. The runner-up was GPT-6 Astra at 43.17%, roughly 7 percentage points behind, while Gemini-3.8-flash scored 32.71%, Qwen3.8-max scored 28.31%, and Claude-Fable-5.1 came in at 25.22%.”

Dai did note that Glimpse’s results “come from a single standalone run of the model itself, without any agents or tool harnesses,” whereas the six models in the paper were averaged across three runs under their reported configurations.

Image 6

Results of frontier models on the DecoyCount benchmark, averaged over 3 runs each.

So, what does DecoyCount uncover that existing benchmarks miss? According to Dai, it jointly tests both vision and reasoning—whereas most benchmarks tend to conflate the two.

“When a human sees an apple in front of a mirror, we know there’s only one apple and the other is just a reflection. But a model might miss the apple entirely (a vision problem), or count both the apple and its reflection as two (a reasoning problem). Most benchmarks that test for these errors only verify the final number, treating both failures identically. If you’re trying to solve the underlying problem, not knowing whether it’s a vision issue or a judgment issue becomes a real challenge.”

Image 7

Frontier model comparison from the DecoyCount paper

What Exactly Is Visual Thinking?

Andrew Dai is by no means a newcomer to the AI field. Back in 2015 during his time at Google, he co-authored a paper titled [Semi-supervised sequence le

#播客#深度访谈#4Allapi.com

Published by the 4All API team

Original link:https://latent.space/p/elorian/

Need a mainstream LLM API? 4All API gives you one key to call OpenAI, Anthropic, Google Gemini, Qwen, DeepSeek, and dozens more — at official-pass-through pricing, with enterprise-grade reliability, integrated in 5 minutes.

Sign up for the 4All API console →