pipette
ESEspañol

Training-Free Prompt Selection for Efficient Reinforcement Learning With Partial Rollouts

Lei Jiang, Honghui Bao, Yue Zhou, Zhengyao Gu, Langzhou He, Natalie Parde

Preprint

In the authors' words

Reinforcement learning with verifiable rewards (RLVR) has emerged as a prominent paradigm for training reasoning models, but its computational cost makes efficient data selection increasingly important. Since training examples contribute unequally to learning, selecting a small, informative subset can offer a more efficient alternative to using the entire data pool. The training-free selection method estimates prompt value with hidden states from complete greedy rollouts. However, in our datasets, -- of these rollouts fail to terminate within the generation budget. We introduce LIFT, a training-free data selector that identifies impactful prompts using only partial reasoning rollouts. At its core, LIFT uses Partial Rollout Context Lift (PRCL), a score inspired by pointwise -information that accumulates token-level log-likelihood ratios between a question-conditioned context and a null context with the question removed. This enables prompt scoring from a short rollout prefix rather than a complete trajectory. Across Qwen2.5-Math-1.5B and DeepSeek-R1-Distill-1.5B, prefixes retaining at least 90% and 95% of the full-rollout PRCL score, respectively, require only 26% and 4.7% of the tokens generated by full rollouts. Compared with SHIFT, LIFT reduces selection cost by 3.7--9.1 while achieving competitive downstream RLVR performance, with the highest six-benchmark average in three of six settings.

Main resultThe abstract does not state a limitation.

Appeared: Tuesday, October 20. arXiv. Preprint, not yet peer-reviewed.