pith. sign in

arxiv: 2605.03065 · v4 · pith:PCNMERUMnew · submitted 2026-05-04 · 💻 cs.LG · cs.RO

OGPO: Sample Efficient Full-Finetuning of Generative Control Policies

Sarvesh Patil , Mitsuhiko Nakamoto , Manan Agarwal , Shashwat Saxena , Jesse Zhang , Giri Anantharaman , Cleah Winston , Chaoyi Pan
show 9 more authors
Douglas Chen Nai-Chieh Huang Zeynep Temel Oliver Kroemer Sergey Levine Abhishek Gupta Hongkai Dai Paarth Shah Max Simchowitz
This is my paper

Pith reviewed 2026-07-01 00:00 UTC · model grok-4.3

classification 💻 cs.LG cs.RO
keywords generative control policiesoff-policy optimizationrobot manipulationpolicy finetuningdiffusion policiesPPO objectivebehavior cloningcritic stabilization
0
0 comments X

The pith

OGPO finetunes generative control policies from poor starts by propagating gradients through full sampling with off-policy critics.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces OGPO to make full finetuning of generative control policies practical and sample-efficient. It keeps off-policy critic networks to reuse past data and modifies the PPO objective so that gradients flow all the way through the policy's generative sampling steps, with the critics supplying the terminal reward signal. This setup lets the method reach strong results on robot manipulation benchmarks and, uniquely, recover near-complete task success from badly initialized behavior-cloning policies even when the replay buffer contains no expert trajectories. The authors also supply three stabilization techniques that keep the critics from being over-exploited in both state and pixel observations.

Core claim

OGPO is an off-policy algorithm for finetuning generative control policies that maintains critic networks to maximize data reuse and propagates policy gradients through the full generative sampling process via a modified PPO objective that uses the critics as terminal reward. It reaches state-of-the-art performance on multi-task manipulation, high-precision insertion, and dexterous control, and is the only method shown to fine-tune poorly initialized behavior cloning policies to near full task success with no expert data in the replay buffer and minimal hyperparameter tuning.

What carries the argument

The modified PPO objective paired with off-policy critic networks that act as terminal rewards, allowing gradient flow through the entire generative sampling chain of the policy.

If this is right

  • Generative control policies can be driven from low to near-full task success using only online data and no expert demonstrations in the replay buffer.
  • The same algorithm produces state-of-the-art results across multi-task, high-precision insertion, and dexterous manipulation without extensive per-task hyperparameter search.
  • Stabilization techniques such as success-buffer regularization, two-sided conservative advantages, and Q-variance reduction are required to prevent critic over-exploitation in both state-based and pixel-based settings.
  • OGPO outperforms prior alternatives specifically on the sub-problems of policy steering and learning residual corrections.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same gradient-propagation idea could be tested on generative policies outside robotics, such as sequence or image generation models that must be steered by scalar rewards.
  • If the critic-stability tricks generalize, they may reduce the amount of online data needed in other off-policy reinforcement-learning settings that use long-horizon generative policies.
  • A natural next measurement would be how far the method can push performance when the initial behavior-cloning policy is even weaker than those tested here.

Load-bearing premise

Off-policy critic networks can be kept stable while policy gradients are back-propagated through the full generative sampling process without creating new instabilities or needing heavy per-task retuning.

What would settle it

An experiment in which a poorly initialized behavior-cloning policy fails to reach near-full task success after OGPO finetuning on a standard multi-task or insertion benchmark, or in which success rates fall below existing baselines once the listed stabilization tricks are removed.

Figures

Figures reproduced from arXiv: 2605.03065 by Abhishek Gupta, Chaoyi Pan, Cleah Winston, Douglas Chen, Giri Anantharaman, Hongkai Dai, Jesse Zhang, Manan Agarwal, Max Simchowitz, Mitsuhiko Nakamoto, Nai-Chieh Huang, Oliver Kroemer, Paarth Shah, Sarvesh Patil, Sergey Levine, Shashwat Saxena, Zeynep Temel.

Figure 1
Figure 1. Figure 1: OGPO enables sample-efficient full-policy finetuning of generative control policies. Left: A GCP induces a denoising MDP inside each environment step. OGPO severs the bi-level MDP at the executed action and uses an off-policy critic as a terminal reward for PPO-style optimization over purely computational denoising trajectories. Middle: This off-policy policy extraction substantially improves sample effici… view at source ↗
Figure 2
Figure 2. Figure 2: We recall the bi-level MDP from [Ren et al., 2024], which embeds action-level trajectories into the environmental dynamics. OGPO truncates this MDP at the end of each denoising trajectory, using Q-values as a terminal, action-trajectory-level reward. This enables off-policy policy extraction via on-policy policy optimiza￾tion. Our starting point is the bi-level MDP formulation adopted from [Ren et al., 2024] ( view at source ↗
Figure 3
Figure 3. Figure 3: Visual depiction of the different off-policy RL algorithms. (left) DSRL trains an initial noise steering policy, while EXPO trains a residual policy to modify the final GCP action. (center) QC drives policy improve￾ment via supervised finetuning (SFT) of Best-of-N actions ranked via the critic, while BPTT backpropagates the gradients sequentially through the entire GCP. (right) OGPO uses an ensemble of cri… view at source ↗
Figure 4
Figure 4. Figure 4: The above plots show the full training comparison between (a) Vanilla OGPO, (b) OGPO+, and (c) OGPO+CA, on state-based ROBOMIMIC tasks. The red axis shows success rate and the blue axis shows the mean length of successful trajectories.By aggressively maximizing sparse reward, OGPO optimizes for both task success rate, and completion in few steps. Without further regularization, the two can be in tension, c… view at source ↗
Figure 5
Figure 5. Figure 5: On ablating actor and critic ob￾servation modalities, we observe that vanilla OGPO fails to improve policy performance from image-based critics. of pixels, we compare four variants: (1) state-based actor/state-based critic; (2) pixel-based actor/state-based critic (3) pixel-based actor/pixel-based (4) state-based actor/pixel-based critic. We plot variants (1-3) in view at source ↗
Figure 6
Figure 6. Figure 6: We perform a small sweep of ab￾lations adding Best-of-N (BoN) Inference and Success Buffer on ROBOMIMICTOOLHANG. [Mark et al., 2024, Dong et al., 2025, Li et al., 2025], using the target critic as verifier. In, OGPO+ we do the same with a slightly modified critic QBON described in Appendix A.1. We remark that, due to the aggressive policy extraction, Best-of￾N inference yields only marginal additional perf… view at source ↗
Figure 7
Figure 7. Figure 7: We take early-, mid-, and late- training checkpoints for OGPO and OGPO+CA to rollout 32 trajectories and visualize the min, mean, and max Q vs Monte-Carlo returns. (a) Shows OGPO’s Q values fluctuating widely between positive and negative values. (b) Shows OGPO+CA’s Q values converging more stably around the y = x axis A second challenge in offline-to-online RL is the pervasive “dip” in performance that ar… view at source ↗
Figure 8
Figure 8. Figure 8: We In principle, the PPO clipping mitigates serves as an approximation for a KL trust-region, which has the intended affect of regularizing the policy to mitigate critic over-exploitation. However, given that the generative policies in OGPOare more expressive than traditional monolothic RL policies, we find above that PPO clipping does not suffice. Instead, we leverage a stronger form of regularization - χ… view at source ↗
Figure 9
Figure 9. Figure 9: Comparison against natural off-policy baselines (EXPO, DSRL, QC), and on-policy algorithms modified to use OGPO-style off-policy advantages (OFPO++, FQL) on ROBOMIMIC SQUARE, TOOLHANG, and TRANSPORT. 0 0.5M 1M 1.5M 2M Steps 0.0 0.2 0.4 0.6 0.8 1.0 Success rate D4RL/kitchen/complete-v2 0 0.5M 1M 1.5M 2M Steps 0.0 0.2 0.4 0.6 0.8 1.0 D4RL/kitchen/mixed-v2 0 0.5M 1M 1.5M 2M Steps 0.0 0.2 0.4 0.6 0.8 1.0 D4RL/… view at source ↗
Figure 10
Figure 10. Figure 10: Comparison against natural off-policy baselines (EXPO, DSRL, QC) on FRANKA-KITCHEN 0 0.1M 0.2M 0.3M 0.4M 0.5M Steps 0.0 0.2 0.4 0.6 0.8 1.0 Success rate AdroitHandDoor-v1 0 0.1M 0.2M 0.3M 0.4M 0.5M Steps 0.0 0.2 0.4 0.6 0.8 1.0 AdroitHandPen-v1 0 0.1M 0.2M 0.3M 0.4M 0.5M Steps 0.0 0.2 0.4 0.6 0.8 1.0 AdroitHandHammer-v1 0 0.1M 0.2M 0.3M 0.4M 0.5M Steps 0.0 0.2 0.4 0.6 0.8 1.0 AdroitHandRelocate-v1 OGPO OG… view at source ↗
Figure 11
Figure 11. Figure 11: Comparison against natural off-policy baselines (EXPO, DSRL, QC) on the AdroitHand suboptimal performance when the base policy’s performance is poor, such as in KITCHEN tasks. Further, by not updating later steps of the GCP, steering struggles on high-precision tasks such as the ADROIT task suite. We also empirically found it to be sensitive to hyperparameters; in some tasks, DSRL performance crashes desp… view at source ↗
Figure 12
Figure 12. Figure 12: OGPO+ substantially improves sample efficiency com￾pared to the on-policy DPPO algorithm. while DPPO treats the entire bi-level MDP as a single MDP to train with on-policy RL. On final success rates across ROBOMIMIC SQUARE and TRANS￾PORT, this off-policy modification re￾sults in DPPO taking ∼ 10× longer to reach the final success rates achieved by OGPO+. Overall, we find that both OGPO and OGPO+ outperfor… view at source ↗
Figure 13
Figure 13. Figure 13: Actions generated by BC policies on critical states in TOOLHANG overlap in the UMAP space and di￾verge as training progresses. OGPO policies generate ac￾tions along narrower variance of successful actions com￾pared to the baselines. We also see that ∇aQ(s, a) nudges the actions away from this variance axis toward more ef￾ficient execution As illustrated in view at source ↗
Figure 15
Figure 15. Figure 15: OGPO exhibits manifold expansion by making the policy more multimodal as well as execution ef￾ficient. We compare against the natural baselines and a no-negative gradient version of OGPO, in (top) sparse reward, (middle) sparse reward with ∆at compensation, and (bottom) early-stage policy sparse reward with ∆at compensation settings. We observe that OGPO policies in sparse reward settings optimize for exe… view at source ↗
Figure 14
Figure 14. Figure 14: As needed by the task in question, generative control policies extracted via OGPO exhibit multimodal￾ity by faithfully aligning Q-functions toward Q ∗ . We empirically show the effects of OGPO along with all the baselines on a Push-T task from [Chi et al., 2023] and elucidate a detailed sum￾mary of the optimization and exploration behav￾ior of OGPO in Appendix G. 6.2 Which Design Decisions Explain the Per… view at source ↗
Figure 16
Figure 16. Figure 16: OGPO comparisons with policy extraction ablations with AWR-FM AW-OGPO and OFPO++ on ROBOMIMIC environments 0.00 0.25 0.50 0.75 1.00 1.25 1.50 Steps 1e6 0.0 0.2 0.4 0.6 0.8 1.0 Success Rate square-mh-low_dim 0.0 0.5 1.0 1.5 2.0 Steps 1e6 0.0 0.2 0.4 0.6 0.8 1.0 tool_hang-ph-low_dim 0.0 0.5 1.0 1.5 2.0 Steps 1e6 0.0 0.2 0.4 0.6 0.8 1.0 transport-mh-low_dim OGPO+ OGPO- ² AW-OGPO- ² OFPO++ view at source ↗
Figure 17
Figure 17. Figure 17: OGPO comparison with policy extraction ablations with AWR-FM, AW-OGPO and OFPO++ on ROBOMIMIC environments As shown in view at source ↗
Figure 18
Figure 18. Figure 18: Comparison to EXPO with Offline-Ratio = 0.5 18 view at source ↗
Figure 19
Figure 19. Figure 19: OGPO with diffusion poli￾cies. OGPO can successfully improve both flow policy and diffusion policy. We predominantly use flow policies due to the faster inference compute Here, as an example, we illustrate its use in diffusion policies. We study this on the SQUARE task, where we pre-train a diffusion policy on the MH dataset and then apply online improvement with OGPO. As shown in view at source ↗
Figure 20
Figure 20. Figure 20: Bi-level (two-layer) MDP construction. Each environment step t is expanded into K inner action￾generation steps indexed by k ∈ {K − 1, . . . , 0}. The environment transitions and rewards occur only at k = 0, while for k > 0 the state is unchanged and the inner action variable is updated. We formulate the bi-level MDP ( view at source ↗
Figure 21
Figure 21. Figure 21: Comparison of ODE-to-SDE correction F.1 Diffusion Policy Policy Optimization (DPPO, Ren et al. [2024] DPPO fine-tunes diffusion policies by applying PPO directly to the bi-level MDP introduced in Ap￾pendix D. In this construction, each inner denoising step induces an explicit (Gaussian) likelihood, enabling standard policy-gradient updates on the full trajectory in MBILEVEL. DPPO then instantiates the PPO… view at source ↗
Figure 22
Figure 22. Figure 22: UMAP plot of OGPO, OGPO+, and OGPO+CA on ROBOMIMIC TOOLHANG and a no-negative grad ablation of OGPO. We show these rollouts in view at source ↗
Figure 23
Figure 23. Figure 23: UMAP plot of OGPO comparison with various policy extraction methods on ROBOMIMIC TOOLHANG H Ablations and Limitations of OGPO/OGPO+ view at source ↗
Figure 24
Figure 24. Figure 24: OGPO - OGPO+ design ablations show that success buffer plays a crucial role in OGPO+’s performance. H.1 BPTT vs OGPO The most direct way to train off-policy RL policies is to perform gradient ascent on the Q-values. Although this works for simpler policy parameterizations like Gaussian [Fujimoto et al., 2018], or Squashed Gaussian [Haarnoja et al., 2018] policies, directly using Q values to sequentially b… view at source ↗
Figure 25
Figure 25. Figure 25: BPTT uses Q-values directly to backpropagate gradients along the entire GCP chain. This results in unstable gradients and poor convergence. In contrast, OGPO uses PPO-style policy gradient loss using Q-functions described Eq. (3.2). This results in stable gradients and sample-efficient convergence. H.2 OGPO v/s OGPO+, with and without GRPO std (σ) GRPO formulation uses group relative advantage computation… view at source ↗
Figure 26
Figure 26. Figure 26: OGPO+ vs OGPO+ with no-negative gradients 42 view at source ↗
Figure 27
Figure 27. Figure 27: OGPO+ comparison with an ablation of simultaneous steering and residual learning baseline: S/R H.5 Policy Extraction Alternatives (AWR, ASPO from FPO) OGPO separates critic learning from policy extraction: after learning Qφ with the TD objective, the actor update only needs a mechanism for increasing the probability of high-advantage actions and decreasing the probability of low-advantage actions. This ma… view at source ↗
Figure 28
Figure 28. Figure 28: We compare OGPO with DSRL and QC on pixel-based observations and natural language guidance tasks from the LIBERO benchmark Task Structure: LIBERO features procedurally generated tasks with natural language instructions. Tasks require understanding spatial relationships and object attributes from both visual and linguistic modalities. Reward Structure: All Libero tasks use sparse rewards: -1 for each non-s… view at source ↗
read the original abstract

Generative control policies (GCPs), such as diffusion- and flow-based control policies, have emerged as effective parameterizations for robot learning. This work introduces Off-policy Generative Policy Optimization (OGPO), a sample-efficient algorithm for finetuning GCPs that maintains off-policy critic networks to maximize data reuse and propagate policy gradients through the full generative process of the policy via a modified PPO objective, using critics as the terminal reward. OGPO achieves state-of-the-art performance on manipulation tasks spanning multi-task settings, high-precision insertion, and dexterous control. To our knowledge, it is also the only method that can fine-tune poorly-initialized behavior cloning policies to near full task-success with no expert data in the online replay buffer, and does so with few task-specific hyperparameter tuning. Through extensive empirical investigations, we demonstrate that OGPO drastically outperforms methods alternatives on policy steering and learning residual corrections, and identify the key mechanisms behind its performance. We further introduce practical stabilization tricks, including success-buffer regularization, two-sided conservative advantages, and Q-variance reduction, to mitigate critic over-exploitation across state- and pixel-based settings. Beyond proposing OGPO, we conduct a systematic empirical study of GCP finetuning, identifying the stabilizing mechanisms and failure modes that govern successful off-policy full-policy improvement.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces Off-policy Generative Policy Optimization (OGPO), an algorithm for sample-efficient full fine-tuning of generative control policies (diffusion- and flow-based) in robotics. It maintains off-policy critic networks, uses a modified PPO objective with the critic as terminal reward to propagate gradients through the full generative sampling process, and introduces three stabilization heuristics (success-buffer regularization, two-sided conservative advantages, Q-variance reduction). The central empirical claims are state-of-the-art performance on multi-task manipulation, high-precision insertion, and dexterous control; the ability to recover near-full task success from poorly-initialized behavior-cloning policies with no expert data in the online replay buffer; and that this is achieved with few task-specific hyperparameter tunings. The work also includes an empirical study identifying stabilizing mechanisms and failure modes for GCP fine-tuning.

Significance. If the reported results and the sufficiency of the three heuristics hold under broader conditions, the work would be significant for advancing sample-efficient off-policy RL with expressive generative policies, especially in settings with weak initializations. The systematic empirical investigation of stabilization mechanisms provides practical value beyond the specific algorithm. The absence of any convergence analysis or Lipschitz-style argument for the heuristics, however, limits the strength of the generalizability claims.

major comments (2)
  1. [Abstract and stabilization heuristics description] The headline uniqueness and 'few task-specific hyperparameter tuning' claims rest on the empirical sufficiency of the three stabilization heuristics when back-propagating through the full generative chain. No section provides a sensitivity analysis, ablation on hyperparameter ranges across tasks, or argument showing why success-buffer regularization, two-sided conservative advantages, and Q-variance reduction remain sufficient when generative process length, noise schedule, or task distribution changes. This is load-bearing for both the 'only method' assertion and the low-tuning claim.
  2. [Empirical investigations and stabilization section] The manuscript supplies no convergence bound, Lipschitz-style argument, or even empirical monitoring of critic stability (e.g., Q-value divergence or advantage variance) when gradients flow through the entire diffusion/flow sampling process. The stress-test concern about off-policy critic over-exploitation therefore remains unaddressed at the level required to support the stability precondition for the reported results.
minor comments (1)
  1. The abstract states that OGPO 'drastically outperforms method alternatives on policy steering and learning residual corrections,' but the main text should include a dedicated comparison table or figure with exact baselines, number of seeds, and statistical significance to make this claim verifiable.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the constructive review and for identifying areas where additional empirical support can strengthen the manuscript. We address each major comment below with clarifications and commitments to revisions. The work remains primarily empirical, so we focus on expanding the reported analyses rather than adding theoretical results.

read point-by-point responses

  1. Referee: [Abstract and stabilization heuristics description] The headline uniqueness and 'few task-specific hyperparameter tuning' claims rest on the empirical sufficiency of the three stabilization heuristics when back-propagating through the full generative chain. No section provides a sensitivity analysis, ablation on hyperparameter ranges across tasks, or argument showing why success-buffer regularization, two-sided conservative advantages, and Q-variance reduction remain sufficient when generative process length, noise schedule, or task distribution changes. This is load-bearing for both the 'only method' assertion and the low-tuning claim.

    Authors: We agree that the sufficiency of the heuristics is central and that additional sensitivity evidence would strengthen the low-tuning claim. In the revision we will add a new subsection with sensitivity analysis varying the key hyperparameters of all three heuristics (regularization coefficient, advantage clipping bounds, variance reduction scale) over wide ranges on every task. We will also report ablations under altered diffusion/flow step counts and noise schedules. These results will be used to qualify the abstract claim more precisely while retaining the observation that the same hyperparameter set succeeded across the evaluated task suite. We do not claim the heuristics are provably sufficient for arbitrary changes in generative process length or task distribution. revision: yes

  2. Referee: [Empirical investigations and stabilization section] The manuscript supplies no convergence bound, Lipschitz-style argument, or even empirical monitoring of critic stability (e.g., Q-value divergence or advantage variance) when gradients flow through the entire diffusion/flow sampling process. The stress-test concern about off-policy critic over-exploitation therefore remains unaddressed at the level required to support the stability precondition for the reported results.

    Authors: We acknowledge that the manuscript contains no convergence analysis or Lipschitz argument; this is a genuine limitation of the current empirical focus and we do not plan to add such theory. However, we will add new figures that explicitly monitor critic stability (Q-value histograms, advantage variance, and critic loss curves) throughout training for the main experiments, directly addressing the over-exploitation stress test. These plots will demonstrate that the three heuristics keep the critic from diverging in the reported settings. We maintain that the systematic empirical identification of failure modes already provides practical value, even absent theoretical bounds. revision: partial

standing simulated objections not resolved
  • Providing a formal convergence bound or Lipschitz-style argument for the stabilization heuristics

Circularity Check

0 steps flagged

No circularity: empirical algorithm with independent experimental validation

full rationale

The paper introduces OGPO as an empirical algorithm combining off-policy critics with a modified PPO objective for generative policies. All central claims (SOTA performance, ability to fine-tune from poor BC initializations without expert data, few hyperparameter tunings) rest on reported experiments across manipulation tasks rather than any derivation that reduces to fitted inputs or self-citations by construction. Stabilization tricks are presented as practical heuristics whose sufficiency is shown via ablation studies, not assumed a priori. No equations or uniqueness theorems are invoked that collapse to the method's own outputs.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters, axioms, or invented entities are specified in the abstract.

pith-pipeline@v0.9.1-grok · 5830 in / 1085 out tokens · 40192 ms · 2026-07-01T00:00:00.384983+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 40 canonical work pages · 26 internal anchors

  1. [1]

    Stochastic Interpolants: A Unifying Framework for Flows and Diffusions

    Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  2. [2]

    Bengio, P

    Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5 0 (2): 0 157--166, 1994. doi:10.1109/72.279181

    work page doi:10.1109/72.279181 1994

  3. [3]

    Training Diffusion Models with Reinforcement Learning

    Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  4. [4]

    $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. pi_0 : A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  5. [5]

    Large Language Monkeys: Scaling Inference Compute with Repeated Sampling

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher R \'e , and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  6. [6]

    Randomized ensembled double q-learning: Learning fast without a model

    Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021

    work page arXiv 2021

  7. [7]

    Diffusion Policy: Visuomotor Policy Learning via Action Diffusion

    Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  8. [8]

    EXPO: Stable Reinforcement Learning with Expressive Policies

    Perry Dong, Qiyang Li, Dorsa Sadigh, and Chelsea Finn. Expo: Stable reinforcement learning with expressive policies. arXiv preprint arXiv:2507.07986, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  9. [9]

    IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

    Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures, 2018. URL https://arxiv.org/abs/1802.01561

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  10. [10]

    Implicit reparameterization gradients

    Mikhail Figurnov, Shakir Mohamed, and Andriy Mnih. Implicit reparameterization gradients. Advances in neural information processing systems, 31, 2018

    2018

  11. [11]

    One Step Diffusion via Shortcut Models

    Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models. arXiv preprint arXiv:2410.12557, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    D4RL: Datasets for Deep Data-Driven Reinforcement Learning

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2004

  13. [13]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587--1596. PMLR, 2018

    2018

  14. [14]

    arXiv preprint arXiv:1910.11956 , year=

    Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956, 2019

    work page arXiv 1910

  15. [15]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861--1870. Pmlr, 2018

    2018

  16. [16]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33: 0 6840--6851, 2020

    2020

  17. [17]

    arXiv preprint arXiv:2407.13399 , year=

    Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D Lee, Wen Sun, Akshay Krishnamurthy, and Dylan J Foster. Correcting the mythos of kl-regularization: Direct alignment without overoptimization via chi-squared preference optimization. arXiv preprint arXiv:2407.13399, 2024

    work page arXiv 2024

  18. [18]

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  19. [19]

    Auto-Encoding Variational Bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

    work page internal anchor Pith review Pith/arXiv arXiv 2013

  20. [20]

    arXiv preprint arXiv:2510.14830 (2025)

    Kun Lei, Huanyu Li, Dongjie Yu, Zhenyu Wei, Lingxiao Guo, Zhennan Jiang, Ziyu Wang, Shiyu Liang, and Huazhe Xu. Rl-100: Performant robotic manipulation with real-world reinforcement learning. arXiv preprint arXiv:2510.14830, 2025

    work page arXiv 2025

  21. [21]

    Reinforcement Learning with Action Chunking

    Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking. arXiv preprint arXiv:2507.07969, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  22. [22]

    Flow Matching for Generative Modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  23. [23]

    Libero: Benchmarking knowledge transfer for lifelong robot learning

    Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36: 0 44776--44791, 2023

    2023

  24. [24]

    Flow-GRPO: Training Flow Matching Models via Online RL

    Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv:2505.05470, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  25. [25]

    Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  26. [26]

    What Matters in Learning from Offline Human Demonstrations for Robot Manipulation

    Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Mart\' i n-Mart\' i n. What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv:2108.03298, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  27. [27]

    Policy agnostic rl: Offline rl and online rl fine-tuning of any class and backbone.arXiv preprint arXiv:2412.06685, 2024

    Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Aviral Kumar. Policy agnostic rl: Offline rl and online rl fine-tuning of any class and backbone. arXiv preprint arXiv:2412.06685, 2024

    work page arXiv 2024

  28. [28]

    Flow matching policy gradients.arXiv preprint arXiv:2507.21053, 2025

    David McAllister, Songwei Ge, Brent Yi, Chung Min Kim, Ethan Weber, Hongsuk Choi, Haiwen Feng, and Angjoo Kanazawa. Flow matching policy gradients. arXiv preprint arXiv:2507.21053, 2025

    work page arXiv 2025

  29. [29]

    UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction

    Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  30. [30]

    Steering your generalists: Improving robotic foundation models via value guidance

    Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, and Sergey Levine. Steering your generalists: Improving robotic foundation models via value guidance. Conference on Robot Learning (CoRL), 2024

    2024

  31. [31]

    Self-imitation learning

    Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In International conference on machine learning, pages 3878--3887. PMLR, 2018

    2018

  32. [32]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  33. [33]

    Much ado about noising: Dispelling the myths of generative robotic control,

    Chaoyi Pan, Giri Anantharaman, Nai-Chieh Huang, Claire Jin, Daniel Pfrommer, Chenyang Yuan, Frank Permenter, Guannan Qu, Nicholas Boffi, Guanya Shi, et al. Much ado about noising: Dispelling the myths of generative robotic control. arXiv preprint arXiv:2512.01809, 2025

    work page arXiv 2025

  34. [34]

    Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  35. [35]

    Film: Visual reasoning with a general conditioning layer

    Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  36. [36]

    FAST: Efficient Action Tokenization for Vision-Language-Action Models

    Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-language-action models. arXiv preprint arXiv:2501.09747, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  37. [37]

    Information Theory: From Coding to Learning

    Yury Polyanskiy and Yihong Wu. Information Theory: From Coding to Learning. Cambridge University Press, 2025

    2025

  38. [38]

    Diffusion Policy Policy Optimization

    Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. arXiv preprint arXiv:2409.00588, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  39. [39]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj \"o rn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684--10695, 2022

    2022

  40. [40]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889--1897. PMLR, 2015

    2015

  41. [41]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  42. [42]

    Y ., Snell, C., Greer, J., Wu, I., Smith, V ., Simchowitz, M., and Kumar, A

    Amrith Setlur, Matthew YR Yang, Charlie Snell, Jeremy Greer, Ian Wu, Virginia Smith, Max Simchowitz, and Aviral Kumar. e3: Learning to explore enables extrapolation of test-time compute for llms. arXiv preprint arXiv:2506.09026, 2025

    work page arXiv 2025

  43. [43]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  44. [44]

    Denoising Diffusion Implicit Models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  45. [45]

    Consistency models

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023

    2023

  46. [46]

    Do differentiable simulators give better policy gradients? In International Conference on Machine Learning, pages 20668--20696

    Hyung Ju Suh, Max Simchowitz, Kaiqing Zhang, and Russ Tedrake. Do differentiable simulators give better policy gradients? In International Conference on Machine Learning, pages 20668--20696. PMLR, 2022

    2022

  47. [47]

    Jump-start reinforcement learning

    Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Jos \'e phine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jump-start reinforcement learning. In International Conference on Machine Learning, pages 34556--34583. PMLR, 2023

    2023

  48. [48]

    Steering Your Diffusion Policy with Latent Space Reinforcement Learning

    Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning. arXiv preprint arXiv:2506.15799, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  49. [49]

    Bridgedata v2: A dataset for robot learning at scale

    Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023

    2023

  50. [50]

    Simple statistical gradient-following algorithms for connectionist reinforcement learning

    Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8 0 (3): 0 229--256, 1992

    1992

  51. [51]

    Diffusion models for robotic manipulation: A survey

    Rosa Wolf, Yitian Shi, Sheng Liu, and Rania Rayyes. Diffusion models for robotic manipulation: A survey. Frontiers in Robotics and AI, 12: 0 1606247, 2025

    2025

  52. [52]

    Multilingual Universal Sentence Encoder for Semantic Retrieval

    Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Multilingual universal sentence encoder for semantic retrieval, 2019. URL https://arxiv.org/abs/1907.04307

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  53. [53]

    Flow policy gradients for robot control.arXiv preprint arXiv:2602.02481,

    Brent Yi, Hongsuk Choi, Himanshu Gaurav Singh, Xiaoyu Huang, Takara E Truong, Carmelo Sferrazza, Yi Ma, Rocky Duan, Pieter Abbeel, Guanya Shi, et al. Flow policy gradients for robot control. arXiv preprint arXiv:2602.02481, 2026

    work page arXiv 2026

  54. [54]

    Zhang and M

    Fan Zhang and Michael Gienger. Affordance-based robot manipulation with flow matching. arXiv preprint arXiv:2409.01083, 2024

    work page arXiv 2024

  55. [55]

    Zhang, Daniel Pfrommer, Chaoyi Pan, Nikolai Matni, and Max Simchowitz

    Thomas T Zhang, Daniel Pfrommer, Nikolai Matni, and Max Simchowitz. Imitation learning in continuous action spaces: Mitigating compounding error without interaction. arXiv preprint arXiv:2507.09061, 2025 a

    work page arXiv 2025

  56. [56]

    ReinFlow: Fine-tuning flow matching policy with online reinforcement learning.arXiv preprint arXiv:2505.22094, 2025

    Tonghe Zhang, Chao Yu, Sichang Su, and Yu Wang. Reinflow: Fine-tuning flow matching policy with online reinforcement learning. arXiv preprint arXiv:2505.22094, 2025 b

    work page arXiv 2025

  57. [57]

    Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware

    Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  58. [58]

    Efficient online rein- forcement learning fine-tuning need not retain offline data.arXiv preprint arXiv:2412.07762,

    Zhiyuan Zhou, Andy Peng, Qiyang Li, Sergey Levine, and Aviral Kumar. Efficient online reinforcement learning fine-tuning need not retain offline data. arXiv preprint arXiv:2412.07762, 2024

    work page arXiv 2024