This article is 8,000 words long—it is our collaborative paper on an emerging type of AI evaluation. The paper has also been published in PDF format and can be viewed here.
Abstract: AI models have begun to approach saturation on most mainstream benchmarks. But does this mean they can build and ship a real product, conduct a scientific experiment end-to-end, or navigate government bureaucracy? Researchers have begun testing AI in these real-world scenarios. We call these types of evaluations "open-world evaluations." This paper defines open-world evaluations, surveys the lessons learned so far, and proposes best practices for conducting them.
We also introduce CRUX—a collaborative project of 17 researchers from academia, government, civil society, and industry to regularly evaluate frontier AI capabilities through open-world evaluations. In our first experiment, an AI agent built and published an iOS app to the App Store, making only two mistakes, one of which required human intervention. This gives us a first glimpse of potentially useful capabilities; more importantly, it also provides an early warning for AI-driven app store spam (we disclosed this result to Apple a month before publication).
We hope to conduct similar experiments in other real-world domains to identify early warning signs; this will be one of our primary empirical projects over the coming year.
Authors include: Sayash Kapoor, Peter Kirgis, Andrew Schwartz, Stephan Rabanser, J.J. Allaire, Rishi Bommasani, Magda Dubois, Gillian Hadfield, Andy Hall, Sara Hooker, Seth Lazar, Steve Newman, Dimitris Papailiopoulos, Shoshannah Tekofsky, Helen Toner, Cozmin Ududec, Arvind Narayanan
How should we track and forecast AI capabilities? The mainstream answer in the AI community today is benchmarking. For example, METR's time horizons graph has been used by policy analysts, industry leaders, and institutions studying AI risk to argue that AI capabilities are rapidly advancing.
But benchmarks can both overestimate and underestimate progress. To turn a task into a benchmark, it needs to be precisely defined and automatically verifiable. The problem is that any task precise enough to be benchmarked is also precise enough to be optimized for, meaning AI agents can perform exceptionally well on them. Conversely, low accuracy on a benchmark can stem from incidental failures, such as encountering a CAPTCHA on a website, even if the agent is actually capable of solving the underlying task.
To address these limitations, many researchers are turning to a new form of evaluation: lengthy, complex, real-world evaluations that go beyond benchmarking. Anthropic's Nicholas Carlini used a Claude agent to build a C compiler capable of compiling the Linux kernel. Anthropic and Andon Labs designed a free-form experiment where Claude was tasked with managing a small store in an office. While benchmarks typically consist of dozens of tasks and are evaluated automatically, open-world evaluations often involve only a few samples, usually require human intervention, and are evaluated in an open-ended manner, such as by analyzing agent logs.
It is easy to dismiss these evaluations as unscientific: the sample size for each evaluation is one, and they lack standardization and reproducibility. Despite these limitations, we argue that these evaluations are crucial for gathering evidence about AI capabilities. They can provide early warnings of emerging capabilities, thereby informing efforts to build societal resilience; help evaluators identify blind spots in existing benchmarks; and give businesses a clearer understanding of what tasks AI systems will soon be able to perform, informing strategic decisions about AI. We call them open-world evaluations.
In this article, we conceptualize open-world evaluations, review past cases to identify best practices and pitfalls in conducting them, and introduce CRUX—a project aimed at regularly conducting new open-world evaluations. Here are our key insights:
Open-world evaluations are an important emerging category of AI evaluation. As AI systems become more powerful, evaluations of frontier capabilities must also become more complex. Open-world evaluations are the latest link in a long chain of evaluation methods with increasing complexity. We surveyed 10 prominent open-world evaluations conducted over the past year to summarize best practices and key takeaways.
CRUX (Collaborative Research for Updating AI eXpectations) is our attempt to systematically conduct open-world evaluations. Our team members come from government, academia, and non-profits, many of whom have led open-world evaluations and hold diverse expectations about the future of AI. Our goal is to provide empirical evidence of AI systems' current capabilities, even if these evaluations are currently expensive, while providing early warnings for capabilities that may soon become widespread. We plan to regularly publish new open-world evaluations.
In our first CRUX experiment, we tasked an AI agent with developing and publishing a simple iOS app to the App Store. Many benchmarks test an agent's ability to write code. However, publishing an iOS app involves many other steps: signing the app, publishing a privacy policy on a webpage, filling out Apple's forms