This article is about 8,000 words long — a collaborative paper on an emerging type of AI evaluation. The paper is also available as a PDF here.
Abstract: AI models are beginning to saturate most major benchmarks. But does that mean they can build and ship a real product, carry out a scientific experiment end to end, or navigate a government bureaucracy without getting stuck? Researchers have started testing AI in these kinds of real-world settings. We call these evaluations “open-world evaluations.” This article defines open-world evaluations, summarizes what we have learned so far, and offers best practices for conducting them.
We also introduce CRUX, a collaboration among 17 researchers from academia, government, civil society, and industry that will periodically evaluate frontier AI capabilities through open-world evaluations. In our first experiment, an AI agent built and released an iOS app on the App Store, making only two mistakes, one of which required human intervention. This gave us an early signal of potentially useful capabilities and, more importantly, an early warning sign of AI-driven App Store spam. We disclosed this result to Apple one month before publication.
We hope to conduct similar experiments in other real-world domains to surface warning signs early; this will be one of our main empirical projects over the next year.
Authors: Sayash Kapoor, Peter Kirgis, Andrew Schwartz, Stephan Rabanser, J.J. Allaire, Rishi Bommasani, Magda Dubois, Gillian Hadfield, Andy Hall, Sara Hooker, Seth Lazar, Steve Newman, Dimitris Papailiopoulos, Shoshannah Tekofsky, Helen Toner, Cozmin Ududec, Arvind Narayanan
How should we track and forecast AI capabilities? The prevailing answer in the AI community today is benchmarks. For example, METR’s time horizon graph has been used by policy analysts, industry leaders, and organizations studying AI risk to argue that AI capabilities are improving rapidly.
But benchmarks can both overstate and understate progress. To turn a task into a benchmark, it must be precisely defined and automatically verifiable. The catch is that any task precise enough to be used as a benchmark is also precise enough to optimize for, allowing AI agents to excel on it. Conversely, low benchmark accuracy can also result from incidental failures, like encountering a CAPTCHA on a website, even if the agent would otherwise be capable of solving the underlying task.
To address these limitations, many researchers are turning to a new form of evaluation: longer, messier, real-world evaluations that go beyond benchmarks. Anthropic’s Nicholas Carlini used Claude agents to build a C compiler capable of compiling the Linux kernel. Anthropic and Andon Labs designed an open-ended experiment in which Claude had to run a small office store. Benchmarks typically consist of dozens of tasks that are evaluated automatically, whereas open-world evaluations usually involve only a small number of examples, often require human intervention, and are evaluated in an open-ended way, such as by analyzing agent logs.
It is easy to dismiss such evaluations as unscientific: each evaluation has an effective sample size of 1, and they lack standardization and reproducibility. Despite these limitations, we believe they are important for gathering evidence about AI capabilities. They can provide early warning about emerging capabilities, inform efforts to build societal resilience, help evaluators identify blind spots in existing benchmarks, and give businesses a clearer sense of what tasks AI systems may soon be able to perform, thereby informing strategic decisions about AI. We call them open-world evaluations.
In this article, we conceptualize open-world evaluations, review past examples to distill best practices and common pitfalls, and introduce CRUX, a project aimed at regularly conducting new open-world evaluations. Here are our main takeaways:
Open-world evaluations are an important emerging category of AI evaluation. As AI systems become more capable, the evaluations used to elicit frontier capabilities must also become more sophisticated. Open-world evaluations are the newest addition to this increasingly complex evaluation landscape. We review 10 representative open-world evaluations conducted over the past year to identify best practices and key lessons.
CRUX (Collaborative Research for Updating AI eXpectations) is our attempt to systematically carry out open-world evaluations. The team includes people from government, academia, and nonprofit organizations, many of whom have led open-world evaluations before and hold different expectations about AI’s future. Our goal is to provide empirical evidence about AI systems’ current capabilities, even if such evaluations are currently expensive, while also providing early warning about capabilities that may soon become commonplace. We plan to release new open-world evaluations regularly.
In our first CRUX experiment, we asked an AI agent to develop and publish a simple iOS app to the App Store. Many benchmarks test an agent’s ability to write code. But publishing an iOS app also involves many other steps: signing the app, publishing a privacy policy on the web, filling out Apple’s forms, and completing the review process. We cared less about the agent’s coding ability than about whether it could meet the real-world requirements for shipping an app, so we asked it to build a simple app and complete the iOS App Store submission process.
The agent succeeded after making two mistakes, one of which required human intervention (forgetting where the correct credentials were stored and inventing a fictitious phone number during the App Store review process). Developing and publishing the app cost about $1,000 in total. The app is now live on the iOS App Store. We believe the cost could have been much lower: app development and submission alone took just $25; most of the tokens were spent monitoring the app’s status. We contacted Apple one month before publication to disclose the result of this experiment. App store operators should prepare for spam submissions and review hardening, because they may soon see thousands of agentic app submissions.
How can we improve open-world evaluations? What comes next? To make open-world evaluations more useful, evaluators should be explicit about how much and what kind of human intervention is allowed, make the logs collected while the agent solves the task public, and report what the agent actually did to complete the task by analyzing those logs. In future CRUX projects, we will evaluate automation of AI research and development, AI governance, and many other areas.
Open-world evaluations are an important emerging category of AI evaluation
In this section, we define open-world evaluations and map the emerging landscape of such evaluations to distill what has worked and what the limitations are. We also discuss where open-world evaluations can address some of the blind spots of benchmarks. We believe that as AI systems become more capable, the evaluations used to elicit frontier