Authors: Stephan Rabanser, Sayash Kapoor, Rishi Bommasani, Andrew Schwartz, Arvind Narayanan
At Google's developer conference earlier this week, the company announced its latest model, Gemini 3.5 Flash, alongside a new agentic application, Antigravity 2.0. To showcase the capabilities of this new agentic system, Google claimed that a team of agents built an entire operating system. According to the claim, this feat required only a single prompt, cost about $900 in API fees, and was completed collaboratively by dozens of sub-agents.
Does this mean complex software can now be built by AI at a low cost? Not so fast:
The "single prompt" claim is misleading. The blog post claims that the operating system was built from a single prompt. But halfway through the post, Google reveals that the prompt "ended up being thousands of lines long." How many attempts did it take to generate this prompt? How specific were the instructions given to the agents? Without these crucial details, it is difficult to judge whether the real "secret sauce" is a better model or simply more effort put into prompting. Furthermore, the run was executed on a scaffold1 containing specialized roles, delegation to sub-agents, and an agent designed to detect and prevent cheating. In the release post, Google frames this scaffolding as a product feature. However, we do not know if this scaffolding was overfitted to the task of "building an OS from scratch," or if it would perform equally well on other complex software engineering tasks.
Google's account does not clarify what counts as human intervention. The post mentions that the final run to develop the OS had "no additional human guidance or correction." However, it does not define this standard. The post describes infrastructure used to terminate and restart stuck agents. It also mentions a previous run where the agents appeared to cheat, after which the team added anti-cheating measures and re-ran the task. Yet, the post does not report these trial runs as part of the methodology. It also fails to clarify whether any agents had to escalate issues to humans, whether the final run required any manual restarts, approvals, or fixes, or how many retries occurred before the agents succeeded.
The account does not report any attempts to analyze whether the code written by the agents was written from scratch or copied from existing code on the internet. To Google's credit, the blog post does note that toy operating systems are common undergraduate course projects and that public implementations are easy to find. The post itself raises the concern that the agents might simply be regurgitating existing information rather than building an OS from scratch. However, it does not address this concern—no similarity analysis was conducted, nor was there any log analysis to check if the agents copied existing code. Even in the absence of direct copying, writing an OS might not be particularly difficult for the agents given the likelihood of memorized patterns in the training data, so this says little about the agents' ability to create novel software.
Google has not released the lengthy prompt, the code written by the agents, or the run logs, making it impossible for outsiders to independently evaluate these claims. If the source code or agent logs were public, independent researchers could evaluate the quality of these outputs and answer questions such as whether the agents were copying existing code. The blog post only provides a short video capturing a snapshot of the development progress, along with a high-level narrative of the experiment.
On the other hand, the blog post does provide the exact dollar amount for building the OS ($916.92) and the total token budget (2.6 billion tokens in total). Google deserves credit for providing these numbers, which offer useful context. Many evaluations we have reviewed in the past do not disclose costs at all, making their headline conclusions difficult to compare with other evaluations.
Nonetheless, Google's blog post is essentially a press release. We also recognize that expecting it to meet the rigor of scientific research is unrealistic. This type of evaluation—assessing a long-horizon, real-world task in a single run, with the experimenters narrating what the agents did—has become common. Since many of these are conducted by AI companies themselves, it is easy to understand why one might dismiss this entire category of content as hype.
But doing so would be a mistake. We call this emerging paradigm open-world evaluations and identified this trend in a recent paper (and its accompanying blog post). Crucially, we argue that open-world evaluations require a new set of methodological norms. When executed properly, they can provide valuable perspectives that benchmark-based evaluations cannot.
Google's experiment does add to the growing body of evidence that agents or teams of agents can handle certain types of tasks autonomously or semi-autonomously over long periods, making steady progress without getting stuck or descending into chaos. As we argue in our paper, benchmark evaluations are practically infeasible for these kinds of tasks in many ways, including cost. Therefore, this is an exciting moment for independent evaluators in academia, non-profits, and government to step in and bring rigor and credibility to open-world evaluations—qualities that are often hard to find in AI vendors' own claims.
A scaffold is a layer of code, prompts, and tools built around an AI model to give it the ability to act autonomously, handling things like memory, tool access, and interaction with the environment. For example, Claude Code is a scaffold that enables Anthropic's Claude model to act as a coding agent.