Building a Cluster Scheduler: Prioritizing High-Impact Research While Maintaining Full Occupancy
At Ai2, the AI Infrastructure team is responsible for providing GPU compute across the institute, with an emphasis on supporting large-scale distributed training workloads. We view this mission as a pyramid built from four progressive metrics.
The foundation of the pyramid is availability: how often hardware remains healthy and ready for work. Above that is occupancy: the fraction of available time allocated to a specific workload. Next is impact: how often the most valuable workloads receive resources. At the top of the pyramid is utilization: the proportion of GPU compute actually consumed over the lifespan of a workload.
This post explores how we improved the “impact” of our scheduling decisions. Recently, we decommissioned a priority-based scheduler in favor of a new system featuring GPU time budgets, hierarchical fair-share allocation, and time-slicing contracts. As a result, debates over how much GPU time each research project should receive shifted from a case-by-case operational chore into a transparent executive budgeting process.
Overcommitting
At Ai2, we manage thousands of NVIDIA H100, B200, and B300 GPUs across clusters ranging from 88 to 1,024 GPUs in size. Built specifically for large-scale distributed training of AI models, these clusters serve approximately 150 in-house researchers. Their work spans a broad spectrum of AI domains, including full-lifecycle LLM and VLM training, robotic reinforcement learning (RL) simulations, and post-training for scientific agentic use cases.
Like many research labs, our demand for GPU time vastly outstrips supply. Based on submitted workloads, the number of GPUs requested by outstanding jobs at any given moment is 2 to 3 times what is actually available. In other words, for every available GPU hour in the cluster, 2 to 3 different research workloads are competing against one another.
In the past, we relied on a priority-based scheduler and allowed workloads to opt out of preemption. Each team had a cap on the number of GPUs they could use concurrently for non-preemptible workloads. Preemptible workloads, on the other hand, could burst beyond this limit onto idle GPUs. This strategy led to predictable pathologies. For example, we observed GPU “squatting”—users would launch no-op workloads just so they could immediately jump in when needed. This occurred because researchers found they couldn’t spin up debugging workloads with low enough latency for interactive troubleshooting. We also saw “priority inflation,” where eventually 100% of scheduled tasks were set to “HIGH” priority. This meant lower-priority workloads were completely starved of GPU time. Furthermore, because preemption was optional, our on-call engineers spent a significant portion of their ticketing time negotiating how to gracefully shut down non-preemptible workloads running on nodes with known maintenance issues.
Tragedy of the Commons
When these issues surfaced, we didn’t immediately pinpoint the root cause. Our initial attempts to ensure critical work received GPU time focused on enforcing tighter controls over priority settings, which eventually devolved into bypassing the priority-based scheduler altogether by granting key projects exclusive access to dedicated GPUs. Though we didn’t realize it at first, we had essentially built an ideal laboratory for observing the “tragedy of the commons”: individuals competing for a scarce shared resource, acting in their own self-interest, produced globally suboptimal outcomes and led to the overuse of the underlying resource.
We were by no means the first to observe this phenomenon. Resource allocation is a fascinating field bridging algorithm design, economics, and systems management. Its core challenge lies in the fact that users typically know the value of their jobs better than the organization does, yet they are incentivized to conceal that value—or hoard resources even when doing so hurts overall performance. For instance, in the 2011 paper introducing Dominant Resource Fairness, Ghodsi et al. shared an anecdote about a search company that allocated private machines to jobs only if users guaranteed high utilization. As a result, they quickly discovered that “users were writing dead loops in their code to artificially inflate utilization levels.” Hardware continues to evolve, but the fundamental problems that complicate resource allocation remain unchanged.
Budgets Not Schedules
The textbook solution to the tragedy of the commons is privatizing the shared resource—owners are incentivized to maximize the value of their assets. When we allocated exclusive access to pools of GPUs for specific teams, we were effectively practicing a variation of this model, but the granularity was far too coarse. Research inherently comes in waves, which led to GPUs frequently sitting idle. Because teams became ready to run experiments and training at different times, granting exclusive reservations inevitably meant periods where no work was running while another team was starved for compute.
We were essentially solving a manual knapsack problem, trying to cram dynamic research demands into a static schedule. We wanted the aligned incentives that asset ownership provides, but without sacrificing full GPU occupancy.
We decided to evolve the ownership model. Instead of handing out GPU hardware directly to teams, we began allocating shares of GPU time. Predicting future demand requires foreseeing the outcomes of cutting-edge scientific experiments—an impossible feat. However, determining relative priority across research efforts is a strategic matter that can be discussed and decided in advance much more easily. Rather than trying to brute-force a scheduling maze, we empowered leadership to think like investors: deciding how much GPU time to “fund” each research initiative based on its perceived potential impact before workloads are even created. The scheduler could then leverage this information when prioritizing incoming workloads.
Built on this concept, we designed a hierarchical system where managers can proportionally allocate GPU time to the projects and researchers under their purview. As illustrated below, this translates scientific strategy directly into guaranteed shares of GPU time. Regardless of how many jobs are queued elsewhere, Project A1 knows it is entitled to 35% of the cluster’s aggregate capacity.
Values in parentheses represent the percentage of total cluster capacity allocated to leaf projects.
In this system, any request for GPU time must be funded by a budget, or it will receive no preemption protection. Under the legacy system, “HIGH” priority came at zero cost, and non-preemptibility meant that a team


