Today, we are officially releasing Old Ideas, Novel Problems, a new study investigating how reliably large language models (LLMs) evaluate the novelty of scientific research ideas. Alongside the paper, we are releasing a benchmark, dataset, and code for testing novelty judges, as well as an automated pipeline that researchers can use to construct new evaluation suites.
At Ai2, we have been building AI systems designed to support different stages of scientific discovery. DataVoyager helps researchers explore and analyze structured data, AutoDiscovery generates and validates hypotheses, and Theorizer synthesizes evidence across vast literatures into candidate scientific theories. Together with Asta, these projects form an interconnected suite of systems that assist researchers throughout the full research lifecycle—from understanding prior work to formulating, testing, and refining new ideas.
However, generating research ideas is only part of the challenge in automating scientific discovery. Those ideas must also offer genuinely new contributions, meaning these systems need the ability to distinguish truly novel directions from those that mostly repackage existing work.
This evaluation is often delegated to another LLM—what we refer to as a “novelty judge.” Novelty judges assess whether a research idea is novel compared to prior work, either by evaluating an idea in isolation or by comparing two ideas pairwise. Currently, there is no standardized methodology for building such judges: research teams choose their own models, prompts, evaluation formats, reasoning budgets, and sometimes couple them with literature retrieval systems. Each of these choices can substantially sway the final verdict. Yet, these judges are often validated solely on human-authored papers rather than on the AI-generated ideas they are actually deployed to evaluate.
Our benchmark results reveal that novelty evaluation itself is far from reliable. Across tests of 6 LLMs serving as judges (GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6), seemingly minor changes in the evaluation setup led to drastically different judgments on the exact same idea. In one experiment, altering just a few words in the prompt caused a model’s accuracy in identifying which of two ideas was more novel to swing by over 50 percentage points. Incorporating retrieval and increasing reasoning compute did not consistently improve judgments, and two purpose-built novelty evaluation systems actually underperformed a simple prompted Claude Opus 4.6 baseline.
If AI is to play a meaningful role in scientific discovery, we need far more dependable ways to determine whether it is proposing something genuinely new. Our work provides the tools to study this problem while highlighting the many open challenges that remain.
Building a Benchmark for Novelty Judges
To test how reliably novelty judges distinguish between ideas with higher versus lower novelty, we first needed a collection of samples with strong ground-truth evidence regarding the originality of their underlying scientific contributions.
To that end, we built an automated pipeline centered around peer reviews from ICLR 2026. Rather than using paper acceptance as a proxy for novelty, we mined the text of the reviews directly, extracting passages where reviewers explicitly discussed the originality of the contributions. This distinction is crucial: a paper can be accepted despite modest novelty if it is strong in other dimensions; conversely, a rejected paper might contain an original contribution but suffer from other flaws.
We constructed a “high-novelty pool” of 154 human-authored ideas: these were drawn from accepted papers scoring in the top 10% of their respective ICLR tracks, where reviewers uniformly praised their novelty with zero dissenting concerns. Using the inverse criteria, we built a “lower-novelty pool” consisting of 145 ideas from rejected papers in the bottom 10%, where reviewers explicitly challenged their originality and none praised their novelty. In an independent audit of our review-mining pipeline, human annotators agreed with the extracted novelty polarity in 47 out of 50 cases; none of the disagreements changed whether a reviewer was ultimately classified as praising or questioning the paper’s novelty.
Together, these two pools form our first evaluation setup—“Human-Only”—which tests whether novelty judges can differentiate between human-written ideas backed by clear evidence of high or lower novelty. In practice, however, novelty judges are increasingly tasked with assessing AI-generated ideas. We therefore created a second setup: “Human+Generated.”
For this pool, we took the same 1
