Skip to content
Main Site News Console

vLLM V0 to V1: In RL, Prioritize Correctness Before Optimization

· Hugging Face Translated
教程模型卡

vLLM V0 to V1: Correctness Before Corrections in RL

Published on May 6, 2026

Rafael Pardinas rafapi-snow Follow

ServiceNow-AI

Ehsan Kamalloo ehsk Follow

ServiceNow-AI

PipelineRL uses vLLM as the inference engine for rollout generation. The inference engine samples tokens and returns token logprobs; the trainer uses these logprobs to compute policy ratio, KL, clip rate, entropy, and reward. Any difference in how logprobs are computed changes the training dynamics. That is the train–inference mismatch we had to eliminate during the migration from vLLM V0 to V1.

TL;DR. After fixing four things, vLLM V1 aligned with our vLLM V0 reference implementation: processed rollout logprobs, V1-specific runtime defaults, the in-flight weight update path, and fp32 lm_head for the final projection. We fixed backend behavior before changing the RL objective.

The reference run uses vLLM 0.8.5; the V1 run uses vLLM 0.18.1. Figure 1 shows the final result. The red curve is the initial V1 attempt, and the green curve is the final V1 run after the fixes described below.

Image 12

Figure 1. Trainer-side metrics for the vLLM V0 reference version (blue), the initial vLLM V1 attempt (red), and the final vLLM V1 run after fixes (green), including fp32 lm_head. The final V1 run closely tracks V0 on clip rate, KL, entropy, and reward.

Migration Objective

vLLM V1 is a major rewrite of the V0 engine. Because of that, our migration objective was deliberately narrow:

  1. Verify that the rollout logprobs returned by V1 match the format expected by the trainer
  2. Re-run the same workload and compare it against the V0 reference
  3. Evaluate objective-level changes only after backend consistency is restored

The first visible symptoms were:

  • clamp_log_ratio_new_old_indicator
  • kl_new_old
  • entropy
  • reward

These metrics come from a GSPO training run, which is the objective used in this experiment. Similar mismatches can also appear in PPO, GRPO, or any online RL system that treats rollout-side logprobs as part of the optimization objective.

The initial V1 run made the issue obvious. The trainer-side logprobs and rewards diverged from the V0 reference early in training.

Image 13

Figure 2. Current-policy logprob during updates, as computed by the trainer (left), and reward (right). The initial vLLM V1 run (red) gradually separates from the vLLM V0 reference (blue).

The same pattern also showed up in trainer metrics. Clip rate was the easiest signal to read in the initial comparison.

Image 14

Figure 3. Trainer-side metrics for the vLLM V0 reference (blue) and the initial vLLM V1 attempt (red). Clip rate reflects the gap between the rollout and trainer policies; entropy and reward show how that gap propagates through training.

Failure Modes

We grouped the possible causes into three layers:

  1. Semantic mismatch: the backend returns logprobs with different meaning than the trainer expects.
  2. Inference-path mismatch: the backend uses different runtime defaults for caching, scheduling, or request handling, sending the same prompt down a different execution path.
  3. Objective mismatch: the RL objective itself needs to be adjusted for