Skip to content
Main Site News Console

Scaling Git at OpenAI

· OpenAI Translated
OpenAILLM

Scaling Git at OpenAI

Building agent-scale code version control while giving back to the Git ecosystem.

By Aaron Friel, Taylor Blau, and Ted Nyman, Members of Technical Staff

Over the past year, the size of our monorepo and the demands placed on it have grown at an astonishing rate. It quickly became clear that we needed to overhaul our source control system to handle the ultra-high-concurrency workloads driven by Codex.

Quarterly commit growth in OpenAI’s monorepo

Commit growth merged to master relative to Q1 2024

Actual growth

Baseline projection

Without disrupting any engineering work or changing how our teams use Git, we built Pando—a brand-new backend system purpose-built for Git repositories. Today, Pando powers all development activity across OpenAI’s monorepo. Pando was designed with two ambitious goals in mind: first, to deliver horizontally scalable write performance optimized for concurrent pushes across multiple branches; and second, to maximize read throughput so clients can download data fast enough to completely saturate their available network bandwidth.

We achieved these goals while maintaining full compatibility with the Git protocol and Git clients, allowing both our engineers and agents to seamlessly integrate Pando into their existing workflows. In production, Pando has reduced median push latency by nearly 5x. Today, it serves close to 1 PB of Git data daily.

In this post, we’ll share why we decided to build a new source control system and why we chose to maintain Git compatibility. We will also compare this approach to common methods for scaling Git, showing how Pando uniquely addresses OpenAI’s scalability requirements. Finally, we’ll explore our roadmap and how we plan to continue contributing back to upstream Git to support agent-scale workloads.

Why we needed a new source control system

Codex has transformed both the scale and velocity of engineering at OpenAI, empowering engineers to run far more work in parallel than ever before. That means more branches and pushes, more merges, and significantly more fetches and clones across development environments, research jobs, and continuous integration (CI).

These demands compound rapidly. Today, an engineer often has multiple changes queued up for CI builds simultaneously, and each push can trigger several concurrent jobs fetching the exact same commit.

Starting around the end of last year, the underlying systems powering our monorepo began to buckle under the strain. Fetches and pushes slowed to a crawl, merges failed frequently, and system reliability dipped below 90%. Were we hitting the limits of Git itself, or the limits of how we were hosting our monorepo? Git originated in an open-source community exchanging kernel patches over email. For over two decades, as software engineering practices evolved, developers continually adapted their toolchains around Git. When we began hitting bottlenecks in our own monorepo, being able to build on top of existing work was essential for us.

We found a path that preserved this familiar workflow while completely re-architecting the backend to process pushes to different branches in parallel and serve the same history to massive numbers of concurrent readers. Keeping Git as the bedrock of our source control workflow means that improvements made to client tooling benefit not just OpenAI, but have a much broader impact. We look forward to contributing these improvements from our Git fork back upstream to share with the wider community.

Engineering Git for scale

Scaling a source control system involves two core dimensions: supporting high volumes of writers pushing to distinct branches, and serving high volumes of readers fetching identical history. Pando is optimized for both.

First, Pando coordinates updates to individual branches independently. Updates to unrelated branches can proceed in parallel, and readers fetching a specific branch can retrieve the entire history required for that version. The trade-off here is that readers inspecting multiple branches simultaneously might see them at slightly different points in time.

Second, Pando reuses packfiles (Git’s packed file format for compressing multiple objects)…