Ai2 Launches Old Ideas Novel Problems Benchmark for LLM Novelty Judges

Executive Overview

Ai2 (Allen Institute for AI) has released Old Ideas, Novel Problems, an open benchmark, dataset, and codebase designed to evaluate the reliability of large language models (LLMs) when judging the novelty of scientific research ideas. Announced via the official Hugging Face blog, the study is immediately available for free and reveals critical fragility in LLM-based novelty assessment: a minor prompt change caused GPT-5.4 accuracy to plummet from 93.5% to 40.9% (below chance) on Human+Generated data. The research also quantifies the cost of specialized evaluation systems, ranging from $2.59 for a simple model to $82.05 for an AI Scientist pipeline, underscoring the trade-off between cost and reliability in automated scientific review.

Deep-Dive Features & Architecture

The Ai2 team built the benchmark on an automated pipeline that extracts novelty signals from ICLR 2026 reviews rather than relying on paper acceptance as a proxy. The dataset comprises a high-novelty set of 154 human ideas from papers accepted in the top 10% of their field with explicit novelty praise, and a low-novelty set of 145 ideas from rejected papers in the bottom 10% with originality objections. In a validation test, the annotator agreed with the extracted novelty signal polarity in 47 out of 50 cases, and no disagreement would have changed the paper's classification. The team also created a Human+Generated setting pairing 154 human ideas with 154 LLM-generated ideas produced by a simple model without search, tools, or iteration; an independent expert preferred the human idea in all 30 cases reviewed.

The team tested six LLM judges—GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6—across pointwise (absolute rating) and pairwise (comparing two ideas) evaluations. Variables included prompt phrasing, inference budget, availability of related papers, idea formatting, number of aggregated judgments, and generation model. Results showed that the largest performance shifts came from prompt phrasing: when GPT-5.4 was presented with Human+Generated pairs and a prompt stating that reviewers found only one idea novel and asking it to identify which, accuracy reached 93.5%. However, with a simpler prompt merely asking which idea is novel, accuracy dropped to 40.9% (below chance) and judgments flipped for more than half of the same pairs.

On Human-Only data, the same prompt change had an effect of only about 7 percentage points, suggesting that novelty judges may appear stable on human research but behave radically differently on AI-generated ideas. Yet human ideas were not immune: Claude Opus 4.6 accuracy shifted by 21 points on Human-Only. There was no globally best prompt; removing an instruction warning against trivial contributions improved accuracy for some judges and reduced it for others. When comparing ideas from Claude Opus 4.5 and Claude Sonnet 4.5, and from GPT-5.4 and GPT-5.1, between a quarter and a third of judgments changed on average across six judges.

Benchmark & Competitive Performance

The study evaluated specialized novelty assessment systems against simple LLM baselines. The Idea Novelty Checker, a dedicated pipeline, cost $30.66 per evaluation and achieved 0.642 macro-F1, while Sakana AI's AI Scientist cost $82.05 and achieved 0.772 macro-F1. In contrast, a simple Claude Opus 4.6 model with majority voting cost only $2.59 and achieved 0.854 macro-F1, outperforming both specialized systems at a fraction of the cost. The agentic judge GPT-5.6 Sol, equipped with arXiv search, scored 0.78 macro-F1 at 10× the standard cost, failing to surpass the simpler Claude Opus 4.6 (0.89 macro-F1). These results indicate that additional search or reasoning does not consistently improve novelty judgment accuracy and can significantly increase costs.

Industry Impact & Enterprise Adoption

This benchmark exposes the fragility of relying on LLMs as novelty judges for scientific evaluation. For enterprises and research institutions adopting AI-assisted peer review or idea screening, the findings highlight the need for careful prompt design and validation, as minor changes can drastically alter outcomes. The cost disparity between simple and specialized systems suggests that expensive pipelines may not guarantee better performance, urging organizations to benchmark against simple baselines. The open availability of the benchmark, data, and code on Hugging Face enables reproducible research and supports the development of more robust novelty assessment tools. As AI-generated research ideas become more prevalent, reliable novelty detection will be critical for maintaining scientific integrity and guiding investment in innovation.

Conclusion

Ai2's Old Ideas, Novel Problems benchmark provides a rigorous, open framework for evaluating LLM novelty judges, revealing significant prompt sensitivity and cost inefficiencies in specialized systems. The study underscores that current LLMs are not yet reliable arbiters of scientific novelty, especially for AI-generated ideas, and that simple, cost-effective models can outperform complex pipelines. By releasing the benchmark and data freely, Ai2 enables the community to build more trustworthy evaluation methods, a crucial step as AI increasingly participates in scientific discovery.

Media Source: Hugging Face | البيان الرسمي للشركة: المصدر الأصلي | Fact Verification & Analysis: AI Tools Oasis