Control Room: A Central System for Tracking and Aborting Model Workloads in Research Infrastructure
As our models grow increasingly powerful, the associated safety and security risks rise as well, making it ever more critical to detect potential issues and quickly stop models from running when necessary. To address this, we built Control Room.
Control Room is a central system designed to track model workloads across our entire research infrastructure and halt them swiftly if safety or security concerns arise. It connects to every research inference workload, including agentic sampling for model training and evaluation, data processing, and more. Working in tandem with our misalignment monitoring systems, the system can abort or pause a job’s model outputs within 35 seconds of an intervention decision. Its architecture ensures it stays operational even when other systems fail, and if Control Room itself experiences an outage, it automatically halts model outputs.
Control Room has been deployed across the research infrastructure where the need is most acute. By contrast, our external model-serving stack is inherently more centralized and already possesses mature operational capabilities to rapidly take model deployments offline whenever necessary.
How It Works
Control Room operates by integrating several systems: the workload, the inference engine, and the centralized Control Room service.
-
A workload is a running job, such as a reinforcement learning experiment, evaluation, large-scale sampling, or data processing job. We require all research workloads performing agentic inference to be tied to an approved safety case.
-
The inference engine (or simply the engine) is the model itself. You can think of the engine as the model’s brain running directly on GPUs. A workload typically uses one or more models, which may run on one or more distinct inference engines. The same inference engine can also be shared across multiple different workloads.
-
The Control Room service tracks jobs and controls whether they are permitted to continue generating model outputs. Engines check in with the Control Room service before responding to requests. If a job has been aborted, the service denies the request, preventing further output.
Control Room integrates as follows:
-
When launching a new job, the workload code requests a new
job_idfrom Control Room. -
Whenever the workload needs to sample tokens from a model, it sends a request—including the
job_id—to the inference engine running that model. The engine then asks Control Room for permission to process the request, passing along thejob_id. -
Control Room checks whether the job has been terminated; if so, it denies the inference request. The engine then returns a rejected response to the workload’s inference request, halting further model output generation for that job.
Control Room Operational Flow
Research job
<no job ID yet>job_042
Inference engine
Runs models on GPUs
Control Room
- job_039, active
job_039 - job_040, active
job_040 - job_041, active
job_041 - job_042, active
job_042
Step 1: Researcher decides to launch a job
The process begins when a researcher initiates a job—such as an experiment or an evaluation.
Job launches
The job code (or workload code) is owned and executed by the researcher. Inference engines may be launched as part of the job, or standalone engines not tied to any specific job may already be running.
Job registers with Control Room
The workload code requests a job ID, linking the job to an approved safety case. The job itself makes this request.
Control Room returns job ID
Control Room records the job and returns its identifier. The workload will include this ID in its inference requests.
Job requests model output from inference engine
The request includes the job_id. This request is sent directly by the workload code without going through Control Room.
Engine checks permissions
Before processing the request, the inference engine queries Control Room to verify whether the job is still authorized to run.
Control Room permits the request
Because the job remains authorized, Control Room grants permission. The inference engine can now respond to the request.
Engine returns model output
The model generates a response, which the engine sends back to the research job. The job can use the output and continue its work.
An event triggers the decision to abort the job
A human operator or a monitoring system instructs Control Room to abort the job. Control Room marks its ID as stopped. The workload code continues to run.
The job may still attempt to sample model output
We do not rely on the workload code itself to correctly check whether the job is still allowed to run.
Engine checks permissions again
The engine asks Control Room for permission to process the new inference request, passing along the same job ID.
Subsequent authorization is denied
When the engine checks permissions again, Control Room denies the aborted job. The engine must reject the inference request.
Model output stops—job remains alive
The engine returns a rejection response instead of model output. The research process may stay alive, and the engine remains available for other authorized jobs.
Example: Responding to Monitoring Alerts
All our frontier reinforcement learning jobs run under the supervision of monitors. A monitor is another model that inspects everything the model being trained does and raises alerts whenever issues are detected. For instance, our recent DNS-related safety incident was caught by a monitor.
Control Room integrates with our monitoring systems as follows:
-
If the monitoring system detects a potential issue, it determines whether to abort the job based on specific details and severity.
-
In such cases, the monitoring system sends a message to Control Room to abort the job, attaching the
job_id. Control Room records that the job has been terminated. -
The next time a job carrying that
job_idissues an inference request to any model running on any engine to sample more tokens, the request is denied, halting further model output.
Speed and Reliability
The maximum time elapsed between Control Room receiving an instruction to abort a job and the job being completely halted is 35 seconds. This 35-second delay stems from a caching layer for inference authorization, which is required to support high-throughput concurrency. We