pith. sign in

arxiv: 2607.01083 · v2 · pith:3BA52NFJnew · submitted 2026-07-01 · 💻 cs.LG · cs.AI

Staleness-Learning Rate Scaling Laws for Asynchronous RLHF

Jingwei Song , Haofeng Xu , Jie Xiao , Chengke Bao , Jingwei Shi , Pengbin Feng , Weixun Wang , Yuhang Han
show 3 more authors
Chuan Wu Linfeng Zhang Bill Shi
This is my paper

Pith reviewed 2026-07-03 21:49 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords asynchronous RLHFGRPOstalenessscaling lawssurrogate gradient biasstability conditionlearning rate
0
0 comments X

The pith

Stale rollouts in asynchronous GRPO create a per-step surrogate-gradient bias of order O(S * eta) that produces a two-constraint stability condition on the learning rate.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper analyzes the impact of using outdated rollouts when rollout generation and policy updates are decoupled in high-throughput RLHF. It distinguishes the learner's surrogate-gradient mapping from the true total derivative of the population objective and shows, under local boundedness plus distributional and behavior-policy smoothness, that the resulting bias scales linearly with both the maximum rollout lag S and the learning rate eta. From this bias the authors derive a conditional collapse-time scaling law that splits into two regimes: one governed by total learner drift T * eta and one that also depends explicitly on S * eta. The resulting stability bound eta much less than the minimum of two terms involving batch and critical radii explains why maximum stable learning rates can appear only weakly sensitive to staleness when the horizon is limited. A reader would care because the bound supplies an explicit rule for choosing eta that accounts for asynchrony rather than relying solely on empirical search.

Core claim

Under assumptions of local boundedness, distributional smoothness, and behavior-policy smoothness, stale rollouts introduce a per-step surrogate-gradient bias of order O(S * eta), where S denotes the maximum rollout lag and eta denotes the learning rate. The authors further derive a conditional collapse-time scaling law: when within-cycle drift remains below a batch-level clipping radius, collapse is governed primarily by cumulative learner drift T * eta; when the stale-rollout constraint is active, stability instead depends explicitly on S * eta. This yields the two-constraint stability condition eta << min{R_batch / (S * G_upd), R_crit / (T * G_upd)}, explaining why the maximum stable lear

What carries the argument

The per-step surrogate-gradient bias of order O(S * eta) obtained by making the behavior policy explicit in the GRPO surrogate and separating the learner's surrogate mapping from the true total derivative of the distribution-dependent objective.

If this is right

  • When within-cycle drift stays below the batch clipping radius, collapse time is set mainly by the product T * eta.
  • When the stale-rollout constraint binds, the allowable eta shrinks proportionally to 1/S.
  • The two-term minimum supplies an explicit upper bound on eta that incorporates both staleness and total update count.
  • In the horizon-limited regime the bound can make the maximum stable eta appear only weakly dependent on S.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same bias-order analysis could be applied to other surrogate objectives that separate rollout and learner policies to test whether O(S * eta) is generic.
  • If the smoothness conditions are only approximately satisfied, the leading bias term might acquire higher-order corrections that alter the precise form of the stability bound.
  • The scaling law suggests a practical design choice: keep S small enough that the S-dependent term does not dominate the T-dependent term for the target horizon.

Load-bearing premise

The GRPO surrogate satisfies local boundedness together with distributional and behavior-policy smoothness so that the bias order O(S * eta) follows from the lag analysis.

What would settle it

A controlled experiment that measures the actual surrogate gradient bias at fixed eta while varying the maximum rollout lag S and checks whether the observed bias grows linearly with S.

Figures

Figures reproduced from arXiv: 2607.01083 by Bill Shi, Chengke Bao, Chuan Wu, Haofeng Xu, Jie Xiao, Jingwei Shi, Jingwei Song, Linfeng Zhang, Pengbin Feng, Weixun Wang, Yuhang Han.

Figure 1
Figure 1. Figure 1: Staleness–learning-rate sweep on Llama-3.2-1B-Instruct. Columns vary the learning rate (lr = η) and within each panel the curves correspond to staleness S ∈ {8, 16, 32}. Top: training reward; collapse appears as the curve dropping to and staying at zero. Middle: cosine similarity between consecutive update directions (Grad CosSim). Bottom: held-out validation reward. Reading left-to-right, reducing η enlar… view at source ↗
Figure 2
Figure 2. Figure 2: Staleness–learning-rate sweep on Llama-3.2-3B-Instruct, with the same layout as [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Small-learning-rate regime on Llama-3.2-3B-Instruct (η ∈ {8, 7, 6, 5}×10−8 ), shown over a long horizon. Left: training reward; middle: gradient cosine similarity; right: validation reward. Unlike the collapsing runs of Figures 1–2, the Grad CosSim rises only briefly during the initial reward climb and then decays toward zero, where it remains. Near-zero cosine similarity indicates that the zero-mean sampl… view at source ↗
Figure 4
Figure 4. Figure 4: Per-staleness training-reward curves on Llama-3.2-1B-Instruct, separated into indi￾vidual panels for legibility (companion to [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Per-staleness training-reward curves on Llama-3.2-3B-Instruct, separated into individ￾ual panels for legibility (companion to [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
read the original abstract

High-throughput RLHF systems often decouple rollout generation from policy optimization, leading to the use of stale rollouts during learner updates. In this work, we study the effect of such staleness in asynchronous GRPO. We make the behavior policy explicit in the GRPO surrogate objective and distinguish between the surrogate-gradient mapping used by the learner and the true total derivative of a distribution-dependent population objective. Under assumptions of local boundedness, distributional smoothness, and behavior-policy smoothness, we show that stale rollouts introduce a per-step surrogate-gradient bias of order O(S * eta), where S denotes the maximum rollout lag and eta denotes the learning rate. We further derive a conditional collapse-time scaling law: when within-cycle drift remains below a batch-level clipping radius, collapse is governed primarily by cumulative learner drift T * eta; when the stale-rollout constraint is active, stability instead depends explicitly on S * eta. This yields a two-constraint stability condition eta << min{R_batch / (S * G_upd), R_crit / (T * G_upd)}, explaining why the maximum stable learning rate may appear weakly dependent on staleness in the horizon-limited regime.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper analyzes staleness effects in asynchronous GRPO for RLHF. It distinguishes the surrogate-gradient mapping from the true total derivative of a distribution-dependent objective and, under local boundedness, distributional smoothness, and behavior-policy smoothness, derives a per-step surrogate-gradient bias of order O(S * eta). This leads to a conditional collapse-time scaling law and the two-constraint stability condition eta << min{R_batch / (S * G_upd), R_crit / (T * G_upd)}.

Significance. If the derivation is correct and the smoothness assumptions hold for GRPO, the result supplies explicit scaling guidance for choosing learning rates in high-throughput asynchronous RLHF pipelines and clarifies why staleness may appear to have only weak effect on the maximum stable eta in horizon-limited regimes. The explicit separation of surrogate gradient from true derivative is a useful conceptual step.

major comments (2)
  1. [Abstract / derivation of bias term] Abstract (and main derivation): the O(S * eta) bias order and the resulting two-constraint stability condition are obtained only after invoking distributional smoothness and behavior-policy smoothness; the manuscript states these assumptions but supplies neither a verification that they hold for the GRPO surrogate (with its importance-ratio clipping and token-level terms) under nonzero lag S nor a counter-example check when they are violated.
  2. [Abstract / collapse-time scaling law] Abstract: the conditional collapse-time scaling law (within-cycle drift below batch clipping radius vs. stale-rollout constraint active) is presented as following from the three smoothness assumptions, yet no explicit error bounds, Lipschitz constants, or dependence on the clipping radius are given in the abstract; the full derivation must be checked for hidden dependence on fitted parameters.
minor comments (1)
  1. [Abstract] Notation: R_batch, R_crit, G_upd, and T are used in the stability condition without prior definition in the abstract; a short glossary or forward reference would improve readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful reading and the identification of points where the assumptions and derivation details require clarification. We address each major comment below.

read point-by-point responses

  1. Referee: [Abstract / derivation of bias term] the O(S * eta) bias order and the resulting two-constraint stability condition are obtained only after invoking distributional smoothness and behavior-policy smoothness; the manuscript states these assumptions but supplies neither a verification that they hold for the GRPO surrogate (with its importance-ratio clipping and token-level terms) under nonzero lag S nor a counter-example check when they are violated.

    Authors: We acknowledge that the manuscript states the three smoothness assumptions without supplying an explicit verification tailored to the GRPO surrogate (including clipping and token-level terms) under nonzero lag S. The derivation is conditional on these assumptions holding; we view them as standard technical conditions that enable the O(S * eta) bias bound. Because the paper focuses on the derivation under the stated assumptions rather than on empirical or analytic verification for GRPO, we will add a short remark in the revised version clarifying the scope and noting that direct verification or counter-examples lie outside the present scope. This constitutes a partial revision. revision: partial

  2. Referee: [Abstract / collapse-time scaling law] the conditional collapse-time scaling law is presented as following from the three smoothness assumptions, yet no explicit error bounds, Lipschitz constants, or dependence on the clipping radius are given in the abstract; the full derivation must be checked for hidden dependence on fitted parameters.

    Authors: The abstract is a concise summary and therefore omits the explicit Lipschitz constants and error bounds, which appear in the main-text derivation (Section 3). The collapse-time scaling law follows directly from the per-step bias bound together with the definitions of R_batch and R_crit; no additional fitted parameters are introduced. The dependence on the clipping radius is explicit in the stability condition eta << R_batch / (S * G_upd). We will verify that the main text makes all constants and their origins fully transparent so that the derivation can be checked without ambiguity. No revision to the abstract itself is required. revision: no

Circularity Check

0 steps flagged

No circularity; derivation follows from explicit assumptions without reduction to inputs by construction

full rationale

The paper states assumptions (local boundedness, distributional smoothness, behavior-policy smoothness) and derives the O(S * eta) bias order plus the two-constraint stability condition eta << min{R_batch / (S * G_upd), R_crit / (T * G_upd)} by distinguishing the surrogate-gradient mapping from the true total derivative of the population objective. No equations reduce to fitted parameters renamed as predictions, no self-definitional loops, and no load-bearing self-citations or uniqueness theorems imported from prior author work. The central claim is conditional on the listed smoothness conditions and does not collapse to its own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on three domain assumptions invoked to bound the bias; no free parameters or invented entities are introduced in the abstract.

axioms (1)
  • domain assumption local boundedness, distributional smoothness, and behavior-policy smoothness
    Invoked to establish that stale rollouts produce O(S * eta) surrogate-gradient bias and the conditional collapse scaling law.

pith-pipeline@v0.9.1-grok · 5764 in / 1370 out tokens · 26120 ms · 2026-07-03T21:49:33.226565+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages · 7 internal anchors

  1. [1]

    Advances in Neural Information Processing Systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in Neural Information Processing Systems , volume=

    work page

  2. [2]

    Advances in Neural Information Processing Systems , volume=

    Deep reinforcement learning from human preferences , author=. Advances in Neural Information Processing Systems , volume=

    work page

  3. [3]

    arXiv preprint arXiv:2312.14925 , year=

    A survey of reinforcement learning from human feedback , author=. arXiv preprint arXiv:2312.14925 , year=

    work page arXiv

  4. [4]

    Proximal Policy Optimization Algorithms

    Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    DeepSeekMath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Advances in Neural Information Processing Systems , volume=

    Direct preference optimization: Your language model is secretly a reward model , author=. Advances in Neural Information Processing Systems , volume=

    work page

  7. [7]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Advances in Neural Information Processing Systems , volume=

    Hogwild!: A lock-free approach to parallelizing stochastic gradient descent , author=. Advances in Neural Information Processing Systems , volume=

    work page

  9. [9]

    Advances in Neural Information Processing Systems , volume=

    Distributed delayed stochastic optimization , author=. Advances in Neural Information Processing Systems , volume=

    work page

  10. [10]

    Advances in Neural Information Processing Systems , volume=

    Asynchronous parallel stochastic gradient for nonconvex optimization , author=. Advances in Neural Information Processing Systems , volume=

    work page

  11. [11]

    11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14) , pages=

    Scaling distributed machine learning with the parameter server , author=. 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14) , pages=

    work page

  12. [12]

    International Conference on Machine Learning , pages=

    Distributed asynchronous optimization with unbounded delays: How slow can you go? , author=. International Conference on Machine Learning , pages=

    work page

  13. [13]

    Scaling Laws for Neural Language Models

    Scaling laws for neural language models , author=. arXiv preprint arXiv:2001.08361 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 2001

  14. [14]

    Training Compute-Optimal Large Language Models

    Training compute-optimal large language models , author=. arXiv preprint arXiv:2203.15556 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    An Empirical Model of Large-Batch Training

    An empirical model of large-batch training , author=. arXiv preprint arXiv:1812.06162 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    International Conference on Machine Learning , pages=

    Trust region policy optimization , author=. International Conference on Machine Learning , pages=

    work page

  17. [17]

    International Conference on Machine Learning (ICML) , pages=

    Asynchronous Methods for Deep Reinforcement Learning , author=. International Conference on Machine Learning (ICML) , pages=

    work page

  18. [18]

    Massively Parallel Methods for Deep Reinforcement Learning

    Massively Parallel Methods for Deep Reinforcement Learning , author=. arXiv preprint arXiv:1507.04296 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    Espeholt, Lasse and Soyer, Hubert and Munos, Remi and Simonyan, Karen and Mnih, Vlad and Ward, Tom and Doron, Yotam and Firoiu, Vlad and Harley, Tim and Dunning, Iain and Legg, Shane and Kavukcuoglu, Koray , booktitle=

    work page

  20. [20]

    International Conference on Learning Representations (ICLR) , year=

    Distributed Prioritized Experience Replay , author=. International Conference on Learning Representations (ICLR) , year=

    work page

  21. [21]

    International Conference on Learning Representations (ICLR) , year=

    Espeholt, Lasse and Marinier, Rapha. International Conference on Learning Representations (ICLR) , year=

    work page

  22. [22]

    International Conference on Learning Representations (ICLR) , year=

    DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames , author=. International Conference on Learning Representations (ICLR) , year=

    work page

  23. [23]

    arXiv preprint arXiv:2310.00036 , year=

    Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platform , author=. arXiv preprint arXiv:2310.00036 , year=

    work page arXiv