Did Google’s AI Agent Really Build an Operating System for $916?
The Importance of Independent Evaluation
Sayash Kapoor and Arvind Narayanan
May 22, 2026
By Stephan Rabanser, Sayash Kapoor, Rishi Bommasani, Andrew Schwartz, Arvind Narayanan
Just earlier this week at Google I/O, the company announced its latest model, Gemini 3.5 Flash, as well as a new agentic app, Antigravity 2.0. To demonstrate the capabilities of this new agent architecture, Google claimed that a team of agents built a complete operating system. The report said this took just one prompt, cost only about $900 in API fees, and was carried out by dozens of cooperating subagents.
Does this mean complex software can now be built cheaply by AI? Not so fast:
-
The “single prompt” claim is misleading. The blog post says the operating system was built from a prompt. But halfway through the article, Google reveals that the prompt “eventually grew to thousands of lines.” How many attempts did it take to arrive at that prompt? How specific were the instructions given to the agents? Without these key details, it is hard to tell whether the secret sauce was a better model or simply more effort spent prompting the model. In addition, this run was carried out on top of a scaffold1 that included specialized roles, task delegation to subagents, and an agent to detect and prevent cheating. In the launch post, Google presents this scaffold as a product feature. But we do not know whether the scaffold was simply overfit to the task of “build an operating system from scratch,” or whether it would perform equally well on other complex software engineering tasks.
-
Google’s description does not clearly define what counts as human intervention. The article says the final run that developed the operating system “did not have any additional instructions or corrections from humans.” But it does not define that standard. The article describes infrastructure for terminating and restarting agents that got stuck. It also mentions that in a previous run, the agents appeared to be cheating, after which the team added anti-cheating measures and reran the task. But it does not include these pre-runs in its methodological description. The article also does not clearly say whether any agent escalation requests were handled by humans, whether the final run required any manual restarts, approvals, or fixes, or how many retries were needed before the agents finally succeeded.
-
The description does not report whether anyone tried to determine if the agents wrote code from scratch or copied existing code from the internet. In Google’s case, the blog post notes that the toy operating system is a common undergraduate course project, and that public implementations are easy to find. The post itself also raises the concern that the agents may have been parroting information rather than building the OS from scratch. But it does not address this issue — there is no similarity analysis or log analysis to check whether the agents copied existing code. Even without direct copying, writing an operating system may have been relatively easy for the agents because related patterns were memorized in the training data, so this does not tell us much about their ability to produce novel software.
-
Google did not release the lengthy prompt, the code written by the agents, or the run logs, making independent evaluation of the claims impossible. Releasing the source code or agent logs would have allowed independent researchers to assess the quality of the artifacts and answer questions such as whether the agents were copying existing code. The blog post provides only a short video, a snapshot of development progress, and a narrative of the whole experiment.
On the other hand, the blog post does provide the exact cost of building the OS ($916.92), as well as the total token budget (2.6 billion tokens). These are useful context, and we want to credit Google for that. Many of the evaluations we have examined previously do not disclose cost at all, which makes it difficult to compare headline claims across evaluations.
Still, Google’s blog post is, at its core, a press release. We also acknowledge that it is unrealistic to demand full scientific rigor from it. Evaluations like this one — single-run assessments of long-horizon, real-world tasks, narrated by the experimenters describing what the agent did — are becoming quite common. Because many of them are conducted by AI companies themselves, it is easy to dismiss the whole genre as hype.
But that would be a mistake. We call this emerging paradigm open-world evaluations and discussed the trend in a recent paper (and accompanying blog post). The key point is that we think open-world evaluations need a new methodological norm. Done properly, they can provide valuable perspectives that benchmark-based evaluations cannot.
Google’s experiment does offer further evidence that agents, or teams of agents, can work autonomously or near-autonomously for long periods on certain kinds of tasks and make progress without getting stuck or going off the rails. As we argue in the paper, benchmark-based evaluations are in many ways actually impossible for such tasks, including because of cost considerations. That makes this the right moment for independent evaluators from academia, nonprofits, and government to step in and bring the rigor and credibility that are unlikely to be present in AI vendors’ own accounts of open-world evaluations.
A scaffold is a layer of code, prompts, and tools built around an AI model to give it agency and handle things like memory, tool use, and more.