Unlocking Asynchronous Capabilities in Continuous Batching
Published: May 14, 2026
RL: We’ll show how to separate CPU and GPU workloads to achieve major performance gains during inference.
This is the second post in our series on efficient LLM inference. The first post introduced continuous batching from first principles. It covered a few concepts we’ll build on later: KV cache, FlashAttention, attention masks, and more.
On Inference Endpoints, an H200 costs about $5 per hour. That doesn’t sound expensive on an hourly basis, but if you keep it running for a full day, that adds up to $120. In that situation, you want the GPU to be utilized as much as possible.
We’ve already seen that continuous batching improves GPU utilization by scheduling tightly packed batches, avoiding wasted compute on padding. But it doesn’t solve another source of waste: by default, it is synchronous. That means the CPU and GPU take turns working: when the GPU is computing, the CPU waits; when the CPU prepares the next batch, the GPU waits. In a loop that runs hundreds of steps per second, these idle gaps add up, and as we’ll show, they can account for nearly a quarter of total runtime. To keep the GPU busy 100% of the time, we need to eliminate those gaps.
To do that, we can use asynchronous batching: we decouple CPU batch preparation from GPU batch computation so they can run in parallel, keeping the GPU busy at all times 🔥
Synchronous Batching
Here is how naive synchronous batching works:
When the CPU prepares a new batch, it decides which requests to include, updates the KV cache table, evicts requests that finished in the previous round, and admits new requests to fill the freed-up slots. Once that’s done, it transfers the prepared inputs to the GPU. The GPU runs the forward pass and samples, i.e. selects, a new token for each request. The results are sent back to the CPU, which then knows which token each request just generated, and the whole loop repeats.
Notice the red highlight on the right: once the GPU finishes computing, it becomes idle. The next batch cannot start until the CPU has finished its update steps: sampling the output tokens, updating request states, and rescheduling the batch.
That is the core inefficiency of synchronous batching: the CPU and GPU work in alternating turns. When the GPU computes, the CPU is idle; when the CPU updates, the GPU is idle. At no point are they both doing useful work at the same time. For a single forward pass, this may seem like a small cost, but in a continuous batching loop running hundreds of steps per second, those idle gaps accumulate into real throughput loss.
To show this, we analyzed the time spent on CPU and GPU when using an 8B model, a batch size of 32, and generating 8K tokens:
