pith. sign in

arxiv: 2605.13435 · v2 · pith:7DDKR4XUnew · submitted 2026-05-13 · 💻 cs.LG · cs.AI

Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policy

JaeHyeok Doo , Byeongguk Jeon , Seonghyeon Ye , Kimin Lee , Minjoon Seo This is my paper

Pith reviewed 2026-06-30 21:32 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords flow-based policiesreinforcement learningstable optimizationvalue propagationoffline RLexpressive policiesQ-Flowpolicy optimization
0
0 comments X

The pith

Q-Flow enables stable optimization of expressive flow-based policies in reinforcement learning by propagating values along flow dynamics without solver unrolling.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes Q-Flow to resolve the trade-off between stability and expressivity when using flow-based models as policies in reinforcement learning. Flow models offer high capacity but naive gradient optimization requires backpropagating through numerical solvers, causing instability. Q-Flow uses the deterministic flow to propagate terminal trajectory values to intermediate states, allowing stable gradients without unrolling the solver. This approach is tested in offline settings on OGBench, showing average improvements of 10.6 percentage points over baselines and supporting online adaptation.

Core claim

Q-Flow is a framework that leverages the deterministic nature of flow dynamics to explicitly propagate terminal trajectory value to intermediate latent states along the policy-induced flow. This enables stable policy optimization using intermediate value gradients without unrolling the numerical solver, bridging the gap between stability and expressivity in flow-based RL policies.

What carries the argument

The explicit value propagation mechanism along the deterministic flow paths from terminal states to intermediate latent states.

If this is right

  • Flow-based policies can be used at full expressivity without optimization instability.
  • The same framework supports both offline learning and stable online adaptation.
  • Consistent outperformance on challenging offline RL benchmarks by 10.6 percentage points on average.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This value propagation idea could extend to other generative policy models that have deterministic dynamics.
  • It may reduce computational cost in training by avoiding solver unrolling.
  • Could improve sample efficiency in online RL settings.

Load-bearing premise

The deterministic nature of the flow dynamics permits explicit and stable propagation of terminal values to intermediate states without requiring solver unrolling or introducing new instabilities.

What would settle it

Training curves or performance metrics where Q-Flow exhibits instability comparable to or worse than methods that unroll the solver on the OGBench tasks.

Figures

Figures reproduced from arXiv: 2605.13435 by Byeongguk Jeon, JaeHyeok Doo, Kimin Lee, Minjoon Seo, Seonghyeon Ye.

Figure 1
Figure 1. Figure 1: Visualization of 2D datasets, Swiss roll (left) and Two spirals (right). The color indicates the reward of each sample, where the reward increases from dark blue to light green. FBRAC FQL [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of flow-based offline RL methods that utilize gradient-based policy optimization in 2D examples. Re￾sults are shown for the Swiss roll (left two columns) and two spirals (right two columns) environments. Strong BC refers to strong BC regularization, and Weak BC refers to weak BC regularization. 3. The Challenge of Flow-based Policy Optimization in Reinforcement Learning To understand the difficu… view at source ↗
Figure 3
Figure 3. Figure 3: 2D experiment results with Q-Flow. Q-Flow preserves full expressivity while enabling stable policy optimization [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Sample (top) and gradient field (bottom) evolution over the V π ω value landscape in the 2D Swiss roll. Sample distributions are shown in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Flow-consistency of intermediate value. We measure the absolute difference of terminal value and intermediate value along policy-induced flow in 2D Swiss roll environment. intermediate value learning. By setting the regression target with the slowly updating target critic Qϕ¯, we effectively dampen the high variance arising from the shifting flow dynamics, thereby preventing the inner value function from c… view at source ↗
Figure 6
Figure 6. Figure 6: Offline-to-online RL results on the default task in 5 OGBench tasks. Q-Flow consistently outperforms flow-based baselines, demonstrating superior adaptability and stable improvement during online fine-tuning. Results are averaged over 8 seeds, with shaded area indicating 95% bootstrap confidence interval. AM-Giant HM-Medium Antsoccer Cube-Double Puzzle-4x4 Overall 0 20 40 60 80 100 Success Rate (%) 0 0 15 … view at source ↗
Figure 7
Figure 7. Figure 7: Component ablation study on default tasks of 5 OGBench environments. For both studies, we include FBRAC as the default baseline. the BPTT baseline defined as: max θ E τ∼U(0,1) x0∼p0 (s,a=x1)∼D   −V π ω (s, Ψ π τ,0 (x0, s), 0) + αLCFM(θ) | {z } Eq. (5)    . Both methods utilize the learned Intermediate Value func￾tion V π (s, xτ , τ ) to guide the policy, but they differ funda￾mentally in how the poli… view at source ↗
Figure 9
Figure 9. Figure 9: Training cost comparison. We report training step time (ms/step) of flow-based methods in Puzzle-4x4 with different numbers of flow steps. approaches (Janner et al., 2019; Kidambi et al., 2020) to better capture the data distribution. Diffusion and Flow-based offline RL. To model com￾plex, multi-modal behavioral distributions (Chi et al., 2023), recent works have integrated expressive generative models int… view at source ↗
Figure 10
Figure 10. Figure 10: Full 2D Toy Experiment Results. Qualitative comparison of generated samples. Q-Flow consistently captures the multi-modal structure of the target distributions, whereas baselines suffer from mode collapse or divergence. Swiss Roll Two Spirals 8 Gaussians Moons [PITH_FULL_IMAGE:figures/full_fig_p015_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Intermediate Value Landscapes. Visualization of the intermediate value function V π ω (s, xτ , τ ) of Q-Flow with λ = 1 across flow time τ in each 2D distribution, evolving from left (τ = 0) to right (τ = 1). 15 [PITH_FULL_IMAGE:figures/full_fig_p015_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Policy gradient norm over offline RL training across different BC/guidance coefficients (α/λ). BPTT leads to severe optimization instability as BC regularization strength weakens. B.3. Analysis Intermediate Value Analysis [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Full training curves of Q-Flow in OGBench under standard setting. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Full training curves of Q-Flow in OGBench under advanced setting. 0.0 0.2 0.4 0.6 0.8 1.0 Steps (M) 0 20 40 60 80 100 Score Umaze-default 0.0 0.2 0.4 0.6 0.8 1.0 Steps (M) 0 20 40 60 80 100 Umaze-diverse 0.0 0.2 0.4 0.6 0.8 1.0 Steps (M) 0 20 40 60 80 100 Medium-play 0.0 0.2 0.4 0.6 0.8 1.0 Steps (M) 0 20 40 60 80 100 Medium-diverse 0.0 0.2 0.4 0.6 0.8 1.0 Steps (M) 0 20 40 60 80 100 Large-play 0.0 0.2 0.… view at source ↗
Figure 16
Figure 16. Figure 16: We conduct ablation studies on the number of flow steps for the policy network and flow time embedding type for the intermediate value network. D. Additional Results and Ablations D.1. Full Offline RL Results The full offline RL results under standard setting are provided in [PITH_FULL_IMAGE:figures/full_fig_p023_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Absolute value difference across flow timesteps along policy-generated trajectories in each OGBench environment. F. Additional Analysis Intermediate Value Analysis. Here, we provide full intermediate value analysis in the OGBench tasksuite. Concretely, we compute the absolute difference between the terminal value and the intermediate value: |V π ω (s, Ψ π 1,τ (xτ , s), 1) − V π ω (s, xτ , τ )|. To compute… view at source ↗
read the original abstract

There is growing interest in utilizing flow-based models as decision-making policies in reinforcement learning due to their high expressive capacity. However, effectively leveraging this expressivity for value maximization remains challenging, as naive gradient-based optimization requires backpropagating through numerical solvers and often leads to instability. Existing approaches typically address this issue by restricting the expressive capacity of flow-based policies, resulting in a trade-off between optimization stability and representational flexibility. To resolve this, we introduce Q-Flow, a framework that leverages the deterministic nature of flow dynamics to explicitly propagate terminal trajectory value to intermediate latent states along the policy-induced flow. This formulation enables stable policy optimization using intermediate value gradients without unrolling the numerical solver, effectively bridging the gap between stability and expressivity. We evaluate Q-Flow in the offline learning setting on the challenging OGBench suite, where it consistently outperforms state-of-the-art baselines by an average of 10.6 percentage points, while also enabling stable online adaptation within the same framework.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 2 minor

Summary. The paper introduces Q-Flow, a framework for reinforcement learning with flow-based policies. It claims that the deterministic nature of flow dynamics permits explicit propagation of terminal trajectory value to intermediate latent states along the policy-induced flow. This enables stable policy optimization via intermediate value gradients without unrolling the numerical solver, resolving the stability-expressivity trade-off. Empirical evaluation in the offline setting on OGBench shows consistent outperformance of state-of-the-art baselines by an average of 10.6 percentage points, with the same framework supporting stable online adaptation.

Significance. If the propagation mechanism and empirical results hold, the work is significant for enabling expressive flow-based policies in RL without the instability of solver backpropagation. This addresses a core practical barrier in applying generative models to decision-making and could facilitate broader adoption in offline RL. The reported gains on OGBench and the dual offline/online capability add to its potential impact.

minor comments (2)
  1. [Abstract] Abstract: the reported 10.6 percentage point average improvement should specify the underlying metric (e.g., normalized return) and indicate whether results include multiple runs or error bars.
  2. [Abstract] Abstract: the phrase 'challenging OGBench suite' would benefit from a brief parenthetical description of the benchmark's scale or difficulty, or a citation to the original OGBench reference.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive assessment of Q-Flow, the recognition of its significance in addressing the stability-expressivity trade-off for flow-based policies, and the recommendation for minor revision. We are pleased that the propagation mechanism and OGBench results are viewed as potentially impactful.

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper presents Q-Flow as a new framework that exploits deterministic flow dynamics to propagate terminal values explicitly to intermediate states, enabling stable gradients without solver unrolling. No equations, fitted parameters renamed as predictions, or self-citation chains are visible in the provided abstract or description that would reduce the central claim to its own inputs by construction. The performance claims are framed as empirical evaluation on OGBench rather than as the derivation itself. The method is introduced as resolving a stated trade-off via a novel formulation, with no load-bearing steps that collapse to self-definition or prior author results invoked as uniqueness theorems. This is the most common honest finding for a methods paper whose core contribution is a new algorithmic formulation rather than a closed mathematical identity.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only the abstract is available; no information is provided on free parameters, axioms, or invented entities.

pith-pipeline@v0.9.1-grok · 5717 in / 991 out tokens · 26704 ms · 2026-06-30T21:32:51.328451+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 11 canonical work pages · 8 internal anchors

  1. [1]

    Diffusion guidance is a controllable policy improvement operator.arXiv preprint arXiv:2505.23458, 2025

    Frans, K., Park, S., Abbeel, P., and Levine, S. Diffusion guidance is a controllable policy improvement operator. arXiv preprint arXiv:2505.23458,

    work page arXiv

  2. [2]

    D4RL: Datasets for Deep Data-Driven Reinforcement Learning

    Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning.arXiv preprint arXiv:2004.07219,

    work page internal anchor Pith review Pith/arXiv arXiv 2004

  3. [3]

    IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies

    Hansen-Estruch, P., Kostrikov, I., Janner, M., Kuba, J. G., and Levine, S. Idql: Implicit q-learning as an actor- critic method with diffusion policies.arXiv preprint arXiv:2304.10573,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Aligniql: Pol- icy alignment in implicit q-learning through constrained optimization.ArXiv, abs/2405.18187,

    He, L., Shen, L., and Wang, X. Aligniql: Policy alignment in implicit q-learning through constrained optimization. arXiv preprint arXiv:2405.18187,

    work page arXiv

  5. [5]

    Gaussian Error Linear Units (GELUs)

    Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus).arXiv preprint arXiv:1606.08415,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

    Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643,

    work page internal anchor Pith review Pith/arXiv arXiv 2005

  7. [7]

    Reinforcement learning with action chunking

    Li, Q., Zhou, Z., and Levine, S. Reinforcement learning with action chunking. InAdvances in Neural Information Processing Systems, volume 38, pp. 55518–55553, 2025a. Li, Y ., Shao, X., Zhang, J., Wang, H., Brunswic, L. M., Zhou, K., Dong, J., Guo, K., Li, X., Chen, Z., Wang, J., and Hao, J. Generative models in decision making: A survey.arXiv preprint arX...

    work page arXiv

  8. [8]

    Playing Atari with Deep Reinforcement Learning

    Mnih, V ., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning.arXiv preprint arXiv:1312.5602,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    AWAC: Accelerating Online Reinforcement Learning with Offline Datasets

    Nair, A., Gupta, A., Dalal, M., and Levine, S. Awac: Accel- erating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359,

    work page internal anchor Pith review Pith/arXiv arXiv 2006

  10. [10]

    Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

    Park, S., Frans, K., Eysenbach, B., and Levine, S. Ogbench: Benchmarking offline goal-conditioned rl. InInterna- tional Conference on Learning Representations, 2025a. Park, S., Li, Q., and Levine, S. Flow q-learning. InProceed- ings of the 42nd International Conference on Machine Learning, volume 267, pp. 48104–48127, 2025b. 11 Q-Flow: Stable and Expressi...

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  11. [11]

    Behavior Regularized Offline Reinforcement Learning

    Wu, Y ., Tucker, G., and Nachum, O. Behavior regu- larized offline reinforcement learning.arXiv preprint arXiv:1911.11361,

    work page internal anchor Pith review Pith/arXiv arXiv 1911

  12. [12]

    Related Work Offline RL.In offline RL, the primary objective is to maximize the expected return while staying close to the state-action distribution defined by the offline dataset

    12 Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policy A. Related Work Offline RL.In offline RL, the primary objective is to maximize the expected return while staying close to the state-action distribution defined by the offline dataset. This is achieved by training the critic to minimize the Bellman error, and Q- learning is perh...

    2019

  13. [13]

    More approaches include sequence modeling (Chen et al., 2021; Janner et al.,

    or via pessimistic value learning (Kumar et al., 2020). More approaches include sequence modeling (Chen et al., 2021; Janner et al.,

    2020

  14. [14]

    and model-based methods (Janner et al., 2019; Kidambi et al., 2020). Diffusion and Flow-based RL.The application of expressive generative models, such as diffusion and flow models, to RL can be categorized by policy optimization strategies:weighted regression(Peters & Schaal, 2007; Peng et al., 2019; Nair et al., 2020),rejection sampling(Chen et al., 2023...

    2019

  15. [15]

    Rejection sampling-based methods often decouple the value learning and policy extraction

    and QIPO (Zhang et al., 2025). Rejection sampling-based methods often decouple the value learning and policy extraction. When the dataset is provided as in offline RL, they perform in-sample value maximization, such as Implicit Q-learning (IQL; Kostrikov et al. 2021), and use the learned value function to determine the action to be executed: aπ = argmax a...

    2025

  16. [16]

    and IDQL (Hansen-Estruch et al., 2023). While the above two paradigms enjoy the simplicity of application to expressive generative models, they are limited by their reliance on scalar value signals from the critic (Park et al., 2024; Frans et al., 2025). Reparameterized gradient-based methods directly maximize the value of the action generated by the mode...

    2023

  17. [17]

    let the gradient flow through a diffusion process. Since the gradient backpropagation leads to noisy and unstable policy optimization, FQL(Park et al., 2025b) distills the behavioral information of the full flow-based policy to a one-step policy and performs value maximization w.r.t. the one-step policy. While FQL utilizes the reparameterized gradient inf...

    2023

  18. [18]

    However, the fundamental distinction lies in the generative policy class, which dictates optimization complexity and intermediate value construction

    explicitly learns the intermediate value via contrastive energy prediction and is the most similar approach to Q-Flow. However, the fundamental distinction lies in the generative policy class, which dictates optimization complexity and intermediate value construction. Specifically, CEP is built on diffusion policy, i.e., an 13 Q-Flow: Stable and Expressiv...

    2025

  19. [19]

    In contrast, the sparse reward definition used in *-sparse tasks does not award the subtask completion reward and provides the full reward only upon the full completion

    In contrast, manipulation tasks typically involve multiple sequential subtasks (e.g., opening a drawer or toggling a button), resulting in rewards bounded between -Ntask and 0, where Ntask denotes the number of subtasks (up to 16 in the environments tested in this work). In contrast, the sparse reward definition used in *-sparse tasks does not award the s...

    2021

  20. [20]

    It uses a Gaussian policy and serves as the competitive baseline that has been considered state-of-the-art before the adoption of expressive generative models as policies

    is a robust actor-critic baseline that improves upon behavior regularization techniques, such as TD3+BC (Fujimoto & Gu, 2021), through architectural and hyperparameter optimization. It uses a Gaussian policy and serves as the competitive baseline that has been considered state-of-the-art before the adoption of expressive generative models as policies. We ...

    2021

  21. [21]

    Both approaches fall into the class of guidance-based methods, where policy improvement relies on evaluating the outer critic at intermediate latent actions

    is a diffusion-based RL method that aligns the generative model updates with the action-gradient of the critic. Both approaches fall into the class of guidance-based methods, where policy improvement relies on evaluating the outer critic at intermediate latent actions. While these guidance-based methods avoid costly BPTT by directly matching the model pre...

    2023

  22. [22]

    For the policy network, we use Fourier embedding for the flow time embedding

    For policy, we use the Euler method of 10 steps across all tasks. For the policy network, we use Fourier embedding for the flow time embedding. We take the mean of Q ensembles as the default aggregation strategy, or take the minimum for some tasks in thestandard settingas FQL. The aggregation is consistent in the algorithm, i.e., we use the same aggregati...

    2026

  23. [23]

    (∗) denotes the default task per environment

    18 Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policy Table 3.Full offline RL results in OGBench understandard setting.Q-Flow performs comparably or superior to the baselines on most tasks. (∗) denotes the default task per environment. We also include the results of other flow-based RL methods, borrowed from Park et al. (2025b), f...

    2026

  24. [24]

    Here, the results are averaged over 12 seeds following the evaluation protocol considered by Li & Levine (2026). D.2. Additional Ablation Studies We conduct additional ablation studies in the default task of selected OGBench environments understandard setting. The results are averaged over 8 seeds. Flow Steps.Figure 16a compares performance across differe...

    2026

  25. [25]

    For D4RL antmaze evaluation, we borrow the numbers from Lu et al

    for extensive empirical validation of its effectiveness in diverse benchmarks. For D4RL antmaze evaluation, we borrow the numbers from Lu et al. (2023) and Zhang et al. (2025). As in the OGBench experiment, of offline RL experiments, we take 1M offline training steps with a batch size of 256 and report the evaluation result at the last step. For offline-t...

    2023

  26. [26]

    Q-Flow achieves the best overall performance, outperforming prior flow-based methods and remaining competitive with strong diffusion-based baselines

    and QIPO (Zhang et al., 2025), as well as flow-based approaches such as FQL. Q-Flow achieves the best overall performance, outperforming prior flow-based methods and remaining competitive with strong diffusion-based baselines. In particular, Q-Flow matches or exceeds the performance of QGPO and QIPO on several tasks, while demonstrating clear improvements...

    2025