pith. sign in

arxiv: 1506.02438 · v6 · pith:4O4WP36Vnew · submitted 2015-06-08 · 💻 cs.LG · cs.RO· cs.SY· eess.SY

High-Dimensional Continuous Control Using Generalized Advantage Estimation

John Schulman , Philipp Moritz , Sergey Levine , Michael Jordan , Pieter Abbeel This is my paper

Pith reviewed 2026-05-11 04:15 UTC · model grok-4.3

classification 💻 cs.LG cs.ROcs.SYeess.SY
keywords reinforcement learningpolicy gradientscontinuous controladvantage estimationneural networkslocomotiontrust region optimizationvalue functions
0
0 comments X

The pith

Generalized advantage estimation reduces variance in policy gradients for high-dimensional continuous control by exponentially weighting temporal difference residuals.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows how value functions can cut the variance of policy gradient estimates in reinforcement learning while accepting some bias, through an exponentially-weighted advantage estimator similar to TD(lambda). It pairs this with trust region optimization to keep both policy and value function updates stable as data arrives. This combination lets neural networks learn complex behaviors like running gaits for simulated 3D robots directly from raw joint positions and velocities to torques. The approach succeeds on bipedal and quadrupedal locomotion and standing up tasks using only model-free simulation data equivalent to weeks of real time. A reader would care if they want practical ways to apply policy gradients to high-dimensional problems without excessive sample requirements or hand-designed features.

Core claim

We address the first challenge by using value functions to substantially reduce the variance of policy gradient estimates at the cost of some bias, with an exponentially-weighted estimator of the advantage function that is analogous to TD(lambda). We address the second challenge by using trust region optimization procedure for both the policy and the value function, which are represented by neural networks. Our approach yields strong empirical results on highly challenging 3D locomotion tasks, learning running gaits for bipedal and quadrupedal simulated robots, and learning a policy for getting the biped to stand up from starting out lying on the ground. In contrast to a body of prior work,

What carries the argument

The generalized advantage estimator: an exponentially-weighted sum of temporal difference residuals analogous to TD(lambda), which trades bias for lower variance in advantage estimates used by policy gradients.

If this is right

  • Neural network policies map directly from raw kinematics to joint torques without hand-crafted representations.
  • Trust region optimization stabilizes improvement for both policy and value functions despite nonstationary incoming data.
  • Model-free learning succeeds on running gaits for simulated bipeds and quadrupeds plus standing-up tasks.
  • The amount of simulated experience needed corresponds to 1-2 weeks of real time for the biped tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • GAE may apply to other high-variance policy optimization settings such as robotic manipulation or game playing with continuous actions.
  • The bias-variance tradeoff in the estimator could be tuned per-task to optimize sample efficiency beyond the fixed lambda used here.
  • Success in simulation raises the question of whether the same direct mapping from kinematics to torques would transfer to physical robots, though sim-to-real gaps are outside the paper's scope.

Load-bearing premise

A neural network value function approximator can be trained sufficiently accurately to deliver useful advantage estimates without introducing bias that negates the variance reduction.

What would settle it

If the learned policies on the 3D locomotion tasks require sample counts comparable to or higher than high-variance Monte Carlo policy gradients, or fail to produce stable gaits, this would show the variance reduction is not effective in practice.

read the original abstract

Policy gradient methods are an appealing approach in reinforcement learning because they directly optimize the cumulative reward and can straightforwardly be used with nonlinear function approximators such as neural networks. The two main challenges are the large number of samples typically required, and the difficulty of obtaining stable and steady improvement despite the nonstationarity of the incoming data. We address the first challenge by using value functions to substantially reduce the variance of policy gradient estimates at the cost of some bias, with an exponentially-weighted estimator of the advantage function that is analogous to TD(lambda). We address the second challenge by using trust region optimization procedure for both the policy and the value function, which are represented by neural networks. Our approach yields strong empirical results on highly challenging 3D locomotion tasks, learning running gaits for bipedal and quadrupedal simulated robots, and learning a policy for getting the biped to stand up from starting out lying on the ground. In contrast to a body of prior work that uses hand-crafted policy representations, our neural network policies map directly from raw kinematics to joint torques. Our algorithm is fully model-free, and the amount of simulated experience required for the learning tasks on 3D bipeds corresponds to 1-2 weeks of real time.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The paper introduces Generalized Advantage Estimation (GAE), an exponentially-weighted estimator of the advantage function (analogous to TD(λ)) derived from standard returns and value functions, to reduce variance in policy gradient estimates at the cost of bias. It combines GAE with trust-region optimization applied to both policy and value function neural networks for stable learning. Empirical results demonstrate success on challenging 3D locomotion tasks, including learning running gaits for bipedal and quadrupedal robots and standing up from a lying position, using model-free policies that map raw kinematics directly to joint torques, with simulated experience equivalent to 1-2 weeks of real time.

Significance. If the results hold, this provides a practical method for high-dimensional continuous control with neural network policies in model-free RL, addressing variance and non-stationarity issues. Strengths include the first-principles derivation of GAE from RL quantities (returns, value functions) independent of the final performance metric, the combination with trust-region constraints, and the demonstration of complex behaviors without hand-crafted representations. The work advances empirical RL for robotics-like tasks.

major comments (1)
  1. [Experiments] Experiments section: The central claim is that GAE reduces policy-gradient variance enough for learning on 3D locomotion while the bias from the neural-network value function approximator remains tolerable. However, the manuscript provides no direct measurement of advantage-estimate bias or variance on the learned policies, nor an ablation isolating value-function accuracy from the trust-region updates. This leaves unaddressed whether approximation error in the value function negates the variance reduction.
minor comments (2)
  1. [Abstract] The claim that simulated experience corresponds to '1-2 weeks of real time' should be supported by the exact number of timesteps or episodes in the main text or a table for reproducibility.
  2. [Method] The GAE(λ) estimator would benefit from an explicit equation and notation definition in the early sections before the empirical results.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive review and the recommendation for minor revision. The positive assessment of GAE combined with trust-region optimization for high-dimensional continuous control is appreciated. We address the single major comment below.

read point-by-point responses

  1. Referee: Experiments section: The central claim is that GAE reduces policy-gradient variance enough for learning on 3D locomotion while the bias from the neural-network value function approximator remains tolerable. However, the manuscript provides no direct measurement of advantage-estimate bias or variance on the learned policies, nor an ablation isolating value-function accuracy from the trust-region updates. This leaves unaddressed whether approximation error in the value function negates the variance reduction.

    Authors: We agree that the manuscript does not include direct empirical measurements of bias or variance for the advantage estimates under the learned policies, nor an explicit ablation separating value-function approximation quality from the trust-region mechanism. Computing ground-truth advantages is intractable for these tasks without an optimal value function. Our defense of the central claim rests on the observed outcomes: the algorithm learns stable running gaits and stand-up behaviors on 3D bipeds and quadrupeds from raw kinematics, using only model-free experience equivalent to 1-2 weeks of real time. Such complex, high-dimensional policies would be unlikely to emerge if value-function bias dominated or if variance reduction were ineffective. The trust-region updates on both policy and value networks are presented as a joint mechanism for stability rather than isolated components. We will add a clarifying paragraph in the discussion section noting the reliance on end-to-end empirical success and the practical difficulty of direct bias/variance diagnostics in this setting. revision: partial

Circularity Check

0 steps flagged

GAE derivation is self-contained from standard RL definitions

full rationale

The paper derives the exponentially-weighted advantage estimator directly from the definitions of the advantage function A_t = Q_t - V_t and the TD residual delta_t = r_t + gamma V(s_{t+1}) - V(s_t), yielding the standard GAE(lambda) sum without any reduction to fitted parameters, self-citations, or input data by construction. Trust-region policy optimization is referenced separately and does not enter the estimator derivation. No quoted step equates a claimed prediction or result to its own inputs; the method remains falsifiable via external benchmarks on variance reduction and bias in continuous control.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The method rests on standard MDP assumptions and the existence of a sufficiently accurate value function approximator; the only explicit free parameter is the GAE lambda that trades bias for variance.

free parameters (1)
  • lambda
    Exponential weighting factor in the advantage estimator that controls the bias-variance tradeoff; chosen by the practitioner.
axioms (1)
  • domain assumption The environment is a Markov decision process with stationary dynamics.
    Required for the definition of returns and advantage functions used in the estimator.

pith-pipeline@v0.9.0 · 5526 in / 1127 out tokens · 20900 ms · 2026-05-11T04:15:59.942358+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. On the Policy Gradient Foundations of Group Relative Policy Optimization: Credit Assignment, Gradient Sparsity, and Rank Collapse

    cs.LG 2026-06 conditional novelty 8.0

    GRPO's group-mean baseline assigns identical advantages to all tokens under output-only rewards, inducing gradient sparsity and an intrinsic rank-2 structure proven from the zero-sum constraint and confirmed by SVD on...

  2. AtomComposer: Discovering Chemical Space from First Principles with Reinforcement Learning

    cs.LG 2026-05 unverdicted novelty 8.0

    AtomComposer uses online RL with multi-composition training to discover up to 10x more valid 3D isomers on unseen chemical formulas than single-composition baselines.

  3. Agent-BRACE: Decoupling Beliefs from Actions in Long-Horizon Tasks via Verbalized State Uncertainty

    cs.CL 2026-05 unverdicted novelty 8.0

    Agent-BRACE improves LLM agent performance on long-horizon partially observable tasks by 5.3-14.5% through a decoupled belief state of verbalized atomic claims with certainty labels that keeps context length constant.

  4. Exploring Line Bundle Standard Models with Transformers

    hep-th 2026-06 unverdicted novelty 7.0

    A Transformer RL agent is trained to generate valid heterotic line bundle sums on CICYs that satisfy gauge embedding, anomaly cancellation, poly-stability, chirality, and no-exotics constraints.

  5. Mirror Descent Beyond Euclidean Stability: An Exponential Separation in Initialization Sensitivity

    cs.LG 2026-06 conditional novelty 7.0

    Non-quadratic Mirror Descent exhibits exponential initialization sensitivity in convex settings, shown via 3D constructions and KL-regularized simplex examples, with Bregman anchoring proposed for stabilization.

  6. TriSearch: Learning to Optimize Triangulations via Bistellar Flips

    cs.LG 2026-05 unverdicted novelty 7.0

    TriSearch is an RL framework that optimizes triangulations of polytopes using bistellar flips with a circuit-supported subtriangulation action representation, generalizing zero-shot to larger instances and outperformi...

  7. Bridging Atomistic Simulation and Experimental Processing Timescales with Goal-Directed Deep Reinforcement Learning

    cond-mat.mtrl-sci 2026-05 unverdicted novelty 7.0

    An E(3)-equivariant deep RL framework lets an O2 agent discover kinetically plausible diffusion and dissociation pathways in disordered Si/a-SiO2 without hand-crafted reaction coordinates or collective variables.

  8. Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients

    cs.LG 2026-05 unverdicted novelty 7.0

    HPO enables unbiased policy optimization in hybrid action spaces by mixing differentiable simulation gradients with score-function estimates, outperforming PPO as continuous dimensions increase.

  9. Self-Supervised On-Policy Reinforcement Learning via Contrastive Proximal Policy Optimisation

    cs.LG 2026-05 unverdicted novelty 7.0

    CPPO is an on-policy contrastive RL method that derives advantages from contrastive Q-values for PPO optimization, outperforming prior CRL baselines in 14/18 tasks and matching or exceeding reward-based PPO in 12/18 tasks.

  10. Adaptive Smooth Tchebycheff Attention for Multi-Objective Policy Optimization

    cs.RO 2026-05 unverdicted novelty 7.0

    An adaptive smooth Tchebycheff controller for multi-objective RL lets agents reach non-convex Pareto regions in robotic tasks while avoiding the instability of static non-linear scalarizations.

  11. Adaptive TD-Lambda for Cooperative Multi-agent Reinforcement Learning

    cs.LG 2026-05 unverdicted novelty 7.0

    ATD(λ) adapts TD(λ) in MARL via a density ratio estimator on past/current replay buffers to assign λ per state-action pair, yielding competitive or better results than fixed-λ QMIX and MAPPO on SMAC and Gfootball.

  12. From Generic Correlation to Input-Specific Credit in On-Policy Self Distillation

    cs.LG 2026-05 conditional novelty 7.0

    Self-distillation token rewards measure input-response-feedback pointwise mutual information, and CREDIT extracts the input-specific component with contrastive baselines to improve LLM reasoning performance.

  13. Metric-Gradient Projection for Stable Multi-Agent Policy Learning

    cs.LG 2026-05 unverdicted novelty 7.0

    HPML projects multi-agent update fields onto the closest metric-gradient potential flow via Hodge decomposition, yielding Lyapunov potentials and equilibrium-gap bounds.

  14. TuniQ: Autotuning Compilation Passes for Quantum Workloads at Scale for Effectiveness and Efficiency

    quant-ph 2026-05 unverdicted novelty 7.0

    TuniQ uses RL with a dual-encoder, shaped rewards, and action masking to autotune quantum compilation passes, improving fidelity and speed over Qiskit while generalizing across backends and scaling to large circuits.

  15. Controllability in preference-conditioned multi-objective reinforcement learning

    cs.LG 2026-05 unverdicted novelty 7.0

    Standard MORL metrics do not measure whether preference inputs reliably control agent behavior, so a new controllability metric is introduced to restore the link between user intent and agent output.

  16. Omni-scale Learning-based Sequential Decision Framework for Order Fulfillment of Tote-handling Robotic Systems

    cs.RO 2026-05 unverdicted novelty 7.0

    OLSF-TRS is a generalized sequential decision framework using structured combinatorial optimization and multi-agent reinforcement learning for order-tote-robot coordination in tote-handling robotic systems, with near-...

  17. Value-Decomposed Reinforcement Learning Framework for Taxiway Routing with Hierarchical Conflict-Aware Observations

    cs.AI 2026-05 unverdicted novelty 7.0

    CaTR applies value-decomposed RL with hierarchical conflict-aware observations to achieve better safety-efficiency trade-offs than planning, optimization, and standard RL baselines in a realistic airport taxiway simulation.

  18. KL for a KL: On-Policy Distillation with Control Variate Baseline

    cs.LG 2026-05 unverdicted novelty 7.0

    vOPD stabilizes on-policy distillation gradients by subtracting a closed-form per-token negative reverse KL baseline as a detached control variate, preserving unbiasedness while lowering variance and matching expensiv...

  19. Learning Visual Feature-Based World Models via Residual Latent Action

    cs.CV 2026-05 unverdicted novelty 7.0

    RLA-WM predicts residual latent actions via flow matching to create visual feature world models that outperform prior feature-based and diffusion approaches while enabling offline video-based robot RL.

  20. Approximate Next Policy Sampling: Replacing Conservative Target Policy Updates in Deep RL

    cs.LG 2026-05 unverdicted novelty 7.0

    Approximate Next Policy Sampling approximates the next policy's state distribution during training to enable larger safe policy updates in deep RL, demonstrated by SV-PPO matching or exceeding standard PPO on Atari an...

  21. Graph Transformers and Stabilized Reinforcement Learning for Large-Scale Dynamic Routing Modulation and Spectrum Allocation in Elastic Optical Networks

    cs.NI 2026-05 unverdicted novelty 7.0

    A graph transformer with RL stabilizations is the first to exceed benchmarks for dynamic RMSA, supporting up to 13% more traffic load on networks up to 143 nodes.

  22. Graph Transformers and Stabilized Reinforcement Learning for Large-Scale Dynamic Routing Modulation and Spectrum Allocation in Elastic Optical Networks

    cs.NI 2026-05 conditional novelty 7.0

    Graph transformer RL for dynamic RMSA supports up to 13% more traffic than benchmarks on networks up to 143 nodes and 362 links.

  23. Beyond Heuristics: Learnable Density Control for 3D Gaussian Splatting

    cs.CV 2026-05 unverdicted novelty 7.0

    LeGS turns density control in 3D Gaussian Splatting into a learnable RL policy whose reward is derived from a closed-form sensitivity analysis that measures each Gaussian's marginal contribution to reconstruction quality.

  24. Financial Market as a Self-Organized Ecosystem: Simulation via Learning with Heterogeneous Preferences

    q-fin.CP 2026-04 unverdicted novelty 7.0

    Multi-agent reinforcement learning with heterogeneous preferences leads to emergent role specialization whose interactions produce fat-tailed returns and volatility clustering, offering a computational realization of ...

  25. DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation

    cs.CV 2026-04 unverdicted novelty 7.0

    DeVI enables zero-shot physically plausible dexterous control by imitating synthetic videos via a hybrid 3D-human plus 2D-object tracking reward.

  26. EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training

    cs.LG 2026-04 unverdicted novelty 7.0

    EVPO adaptively switches between critic-based and batch-mean advantage estimation using batch-level explained variance to provably achieve no greater variance than the better of PPO or GRPO at every step.

  27. Bounded Ratio Reinforcement Learning

    cs.LG 2026-04 conditional novelty 7.0

    BRRL derives an analytic optimal policy for regularized constrained RL that guarantees monotonic improvement and yields the BPO algorithm that matches or exceeds PPO.

  28. SAT: Sequential Agent Tuning for Coordinator Free Plug and Play Multi-LLM Training with Monotonic Improvement Guarantees

    cs.LG 2026-04 unverdicted novelty 7.0

    SAT trains multi-LLM teams with sequential block updates to deliver monotonic gains and plug-and-play model swaps that provably improve performance bounds.

  29. ScoRe-Flow: Complete Distributional Control via Score-Based Reinforcement Learning for Flow Matching

    cs.RO 2026-04 unverdicted novelty 7.0

    ScoRe-Flow achieves decoupled mean-variance control in stochastic flow matching by deriving a closed-form score for drift modulation plus learned variance, yielding faster RL convergence and higher success rates on lo...

  30. Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning

    cs.LG 2026-04 unverdicted novelty 7.0

    GenAC introduces generative critics with chain-of-thought reasoning and in-context conditioning to improve value approximation and downstream RL performance in LLMs compared to value-based and value-free baselines.

  31. Plasticity-Enhanced Multi-Agent Mixture of Experts for Dynamic Objective Adaptation in UAVs-Assisted Emergency Communication Networks

    cs.MA 2026-04 unverdicted novelty 7.0

    PE-MAMoE combines sparsely gated mixture-of-experts actors with a non-parametric phase controller in MAPPO to maintain plasticity under dynamic user mobility and traffic, yielding 26.3% higher normalized IQM return in...

  32. Cayley Graph Optimization for Scalable Multi-Agent Communication Topologies

    cs.NI 2026-04 unverdicted novelty 7.0

    CayleyTopo uses reinforcement learning to optimize Cayley graph generators for lower diameter, yielding faster and more resilient information flow in multi-agent systems than hand-crafted sparse topologies.

  33. Neural Assistive Impulses: Synthesizing Exaggerated Motions for Physics-based Characters

    cs.AI 2026-04 unverdicted novelty 7.0

    A hybrid neural policy operating in impulse space enables physics-based characters to track exaggerated, dynamically infeasible motions that standard DRL methods cannot stabilize.

  34. A semicontinuous relaxation of Saito's criterion and freeness as angular minimization

    math.AG 2026-04 conditional novelty 7.0

    A new functional S vanishes precisely on free line arrangements and enables discovery of verified free examples for every admissible exponent pair with up to 20 lines.

  35. Bellman Value Decomposition for Task Logic in Safe Optimal Control

    cs.RO 2026-02 unverdicted novelty 7.0

    Bellman values for temporal logic tasks decompose into a graph of reach-avoid, avoid, and reach-avoid-loop equations solved by embedding the graph in a two-layer neural net (VDPPO) for safe high-dimensional control.

  36. SMP: Reusable Score-Matching Motion Priors for Physics-Based Character Control

    cs.GR 2025-12 conditional novelty 7.0

    SMP turns pre-trained motion diffusion models into task-agnostic, reusable reward functions via score distillation sampling, enabling style-specific and composable motion priors for humanoid control without retraining...

  37. AID: Agent Intent from Diffusion for Multi-Agent Informative Path Planning

    cs.RO 2025-12 conditional novelty 7.0

    AID trains diffusion policies via behavior cloning on existing MAIPP planners followed by RL fine-tuning to achieve faster execution and higher information gain in multi-agent coordination.

  38. Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?

    cs.LG 2025-09 conditional novelty 7.0

    Action aliasing from safety projections harms policy-gradient estimates more severely when the projection is inside the policy than when it is outside, but a penalty term restores competitiveness.

  39. Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuning

    cs.LG 2025-08 unverdicted novelty 7.0

    TokenBuncher constrains response entropy via entropy-as-reward RL and a Token Noiser to stop harmful RL fine-tuning while keeping benign performance intact.

  40. Mastering Atari with Discrete World Models

    cs.LG 2020-10 accept novelty 7.0

    DreamerV2 reaches human-level performance on 55 Atari games by learning behaviors inside a separately trained discrete-latent world model.

  41. Dota 2 with Large Scale Deep Reinforcement Learning

    cs.LG 2019-12 accept novelty 7.0

    OpenAI Five achieved superhuman performance in Dota 2 by defeating the world champions using scaled self-play reinforcement learning.

  42. Concrete Problems in AI Safety

    cs.AI 2016-06 accept novelty 7.0

    The paper categorizes five concrete AI safety problems arising from flawed objectives, costly evaluation, and learning dynamics.

  43. Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RL

    cs.LG 2026-07 unverdicted novelty 6.0

    FADE is a self-adapting advantage for policy-gradient RL that reads training dynamics to balance positive/negative gradient mass and difficulty focus, yielding faster peak performance and better accuracy-diversity tra...

  44. ToxiREX: A Dataset on Toxic REasoning in ConteXt

    cs.CL 2026-06 unverdicted novelty 6.0

    ToxiREX is a new dataset of 128k Reddit comments in six languages with hierarchical annotations for implicit toxicity in conversational context based on an existing reasoning schema.

  45. Retroactive Advantage Correction: Closed-Form V-Trace Bias Correction for Delay-Aware RLHF

    cs.LG 2026-06 unverdicted novelty 6.0

    RAC is a closed-form bias correction for delayed rewards in RLHF that is unbiased under full mass reinjection of the delay kernel and reduces to V-trace with no delay.

  46. VibeAct: Vibration to Actions for Contact-Rich Reactive Robot Dexterity

    cs.RO 2026-06 unverdicted novelty 6.0

    VibeAct bridges real vibro-acoustic sensing and sim-based RL via a shared contact/slip representation, outperforming proprioception baselines on contact-rich dexterous tasks with successful real-world transfer.

  47. SpatialFlow-GRPO: Where Spatial Credit Drives Image Editing

    cs.CV 2026-06 unverdicted novelty 6.0

    SpatialFlow-GRPO improves image editing quality by converting region-aware rewards into semantic-region-level optimization signals aligned with latent positions during policy updates.

  48. SpatialFlow-GRPO: Where Spatial Credit Drives Image Editing

    cs.CV 2026-06 unverdicted novelty 6.0

    SpatialFlow-GRPO adds region-level reward feedback and spatial alignment to Flow-GRPO-style RL for image editing, reporting gains on GEdit-Bench, ImgEdit-Bench, and a new MultiEditBench.

  49. Finding the Time to Think: Learning Planning Budgets in Real-Time RL

    cs.LG 2026-06 unverdicted novelty 6.0

    A learned gating policy selects state-dependent planning budgets in variable-delay real-time RL and outperforms fixed-budget and heuristic baselines across Pac-Man, Tetris, Snake, Speed Hex, and Speed Go.

  50. Finding the Time to Think: Learning Planning Budgets in Real-Time RL

    cs.LG 2026-06 unverdicted novelty 6.0

    Trains a gating policy to select state-dependent planning budgets in variable-delay real-time RL, outperforming fixed-budget and heuristic baselines across Pac-Man, Tetris, Snake, Speed Hex, and Speed Go.

  51. Learning with a Single Rollout via Monte Carlo Pass@k Critic

    cs.LG 2026-06 unverdicted novelty 6.0

    SR-PPO trains a Pass@k critic from single-rollout Monte Carlo outcomes to enable token-level advantage estimation in language model RL, yielding stable training and Pass@128 gains on math benchmarks.

  52. Heuresis: Search Strategies for Autonomous AI Research Agents Across Quality, Diversity and Novelty

    cs.AI 2026-06 unverdicted novelty 6.0

    Heuresis evaluates six search strategies for autonomous ML research agents and finds that novel ideas are rare, none rated original, and only one reaches top-10 quality while strategies steer axes but do not expand th...

  53. Heuresis: Search Strategies for Autonomous AI Research Agents Across Quality, Diversity and Novelty

    cs.AI 2026-06 accept novelty 6.0

    Heuresis evaluates six search strategies for LLM research agents and shows they steer ideas along quality-diversity-novelty axes but fail to generate novel ideas that match or exceed known high-performing recipes.

  54. Reinforcement Learning Enables Autonomous Microrobot Navigation and Intervention in Simulated Blood Capillaries

    cs.RO 2026-06 unverdicted novelty 6.0

    RL agents learn chemotactic navigation and flow intervention tasks in a physically detailed blood capillary simulation, discovering parameter-independent strategies and restoring baseline flow without retraining.

  55. LOLLA: Deep Reinforcement Learning for Closed-Loop Link Adaptation Towards a GPU-Accelerated AI-RAN

    eess.SP 2026-06 unverdicted novelty 6.0

    LOLLA is a PPO-based DRL framework that learns continuous SINR offsets for 5G link adaptation, delivering 15-92% throughput gains over OLLA while meeting tunable BLER targets and running with sub-500us latency on GPU-...

  56. Stationary Robust Mean-Field Games under Model Mismatches

    cs.LG 2026-06 unverdicted novelty 6.0

    Develops infinite-horizon stationary robust mean-field games incorporating distributional uncertainty, proves equilibrium existence via fixed-point on contractive Bellman operator, gives convergent algorithm, and deri...

  57. Drowning in Routine: Signal Dilution in Multi-Turn Agent Training

    cs.LG 2026-06 unverdicted novelty 6.0

    Trajectory-level RL methods suffer signal dilution in multi-turn agents scaling as ρ^{-1/2} where ρ is the fraction of decision-relevant turns.

  58. Reinforcement Learning Disrupts Gradient-Based Adversarial Optimization

    cs.LG 2026-06 unverdicted novelty 6.0

    RL training disrupts gradient-based adversarial attacks by inducing unstable low-magnitude gradients that limit the effectiveness of methods like PGD within practical budgets.

  59. PAWS: Preference Learning with Advantage-Weighted Segments

    cs.LG 2026-06 unverdicted novelty 6.0

    PAWS performs policy updates directly using segment-level advantage functions to align utility training with policy optimization in preference-based RL.

  60. Discovering Interpretable Multi-Parameter Control Policies for Evolutionary Algorithms Using Deep Reinforcement Learning

    cs.LG 2026-06 unverdicted novelty 6.0

    Deep RL with action decomposition and reward shifting learns a symbolic multi-parameter policy for (1+(λ,λ))-GA on OneMax that outperforms baselines across problem sizes.

Reference graph

Works this paper leans on

25 extracted references · 25 canonical work pages · cited by 189 Pith papers · 1 internal anchor

  1. [1]

    Neuronlike adaptive elements that can solve difficult learning control problems

    Barto, Andrew G, Sutton, Richard S, and Anderson, Charles W. Neuronlike adaptive elements that can solve difficult learning control problems. Systems, Man and Cybernetics, IEEE Transactions on, 0 (5): 0 834--846, 1983

    work page 1983

  2. [2]

    Reinforcement learning in POMDP s via direct gradient ascent

    Baxter, Jonathan and Bartlett, Peter L. Reinforcement learning in POMDP s via direct gradient ascent. In ICML, pp.\ 41--48, 2000

    work page 2000

  3. [3]

    Dynamic programming and optimal control, volume 2

    Bertsekas, Dimitri P. Dynamic programming and optimal control, volume 2. Athena Scientific, 2012

    work page 2012

  4. [4]

    Convergent temporal-difference learning with arbitrary smooth function approximation

    Bhatnagar, Shalabh, Precup, Doina, Silver, David, Sutton, Richard S, Maei, Hamid R, and Szepesv \'a ri, Csaba. Convergent temporal-difference learning with arbitrary smooth function approximation. In Advances in Neural Information Processing Systems, pp.\ 1204--1212, 2009

    work page 2009

  5. [5]

    Variance reduction techniques for gradient estimates in reinforcement learning

    Greensmith, Evan, Bartlett, Peter L, and Baxter, Jonathan. Variance reduction techniques for gradient estimates in reinforcement learning. The Journal of Machine Learning Research, 5: 0 1471--1530, 2004

    work page 2004

  6. [6]

    Reinforcement learning in feedback control

    Hafner, Roland and Riedmiller, Martin. Reinforcement learning in feedback control. Machine learning, 84 0 (1-2): 0 137--169, 2011

    work page 2011

  7. [7]

    Learning Continuous Control Policies by Stochastic Value Gradients

    Heess, Nicolas, Wayne, Greg, Silver, David, Lillicrap, Timothy, Tassa, Yuval, and Erez, Tom. Learning continuous control policies by stochastic value gradients. arXiv preprint arXiv:1510.09142, 2015

    work page Pith review arXiv 2015

  8. [8]

    Principles of behavior

    Hull, Clark. Principles of behavior. 1943

    work page 1943

  9. [9]

    A natural policy gradient

    Kakade, Sham. A natural policy gradient. In NIPS, volume 14, pp.\ 1531--1538, 2001 a

    work page 2001

  10. [10]

    Optimizing average reward using discounted rewards

    Kakade, Sham. Optimizing average reward using discounted rewards. In Computational Learning Theory, pp.\ 605--615. Springer, 2001 b

    work page 2001

  11. [11]

    An analysis of actor/critic algorithms using eligibility traces: Reinforcement learning with imperfect value function

    Kimura, Hajime and Kobayashi, Shigenobu. An analysis of actor/critic algorithms using eligibility traces: Reinforcement learning with imperfect value function. In ICML, pp.\ 278--286, 1998

    work page 1998

  12. [12]

    On actor-critic algorithms

    Konda, Vijay R and Tsitsiklis, John N. On actor-critic algorithms. SIAM journal on Control and Optimization, 42 0 (4): 0 1143--1166, 2003

    work page 2003

  13. [13]

    Continuous control with deep reinforcement learning

    Lillicrap, Timothy P, Hunt, Jonathan J, Pritzel, Alexander, Heess, Nicolas, Erez, Tom, Tassa, Yuval, Silver, David, and Wierstra, Daan. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    work page internal anchor Pith review arXiv 2015

  14. [14]

    Approximate gradient methods in policy-space optimization of markov reward processes

    Marbach, Peter and Tsitsiklis, John N. Approximate gradient methods in policy-space optimization of markov reward processes. Discrete Event Dynamic Systems, 13 0 (1-2): 0 111--148, 2003

    work page 2003

  15. [15]

    Steps toward artificial intelligence

    Minsky, Marvin. Steps toward artificial intelligence. Proceedings of the IRE, 49 0 (1): 0 8--30, 1961

    work page 1961

  16. [16]

    Policy invariance under reward transformations: Theory and application to reward shaping

    Ng, Andrew Y, Harada, Daishi, and Russell, Stuart. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, volume 99, pp.\ 278--287, 1999

    work page 1999

  17. [17]

    Natural actor-critic

    Peters, Jan and Schaal, Stefan. Natural actor-critic. Neurocomputing, 71 0 (7): 0 1180--1190, 2008

    work page 2008

  18. [18]

    Trust Region Policy Optimization

    Schulman, John, Levine, Sergey, Moritz, Philipp, Jordan, Michael I, and Abbeel, Pieter. Trust region policy optimization. arXiv preprint arXiv:1502.05477, 2015

    work page Pith review arXiv 2015

  19. [19]

    Introduction to reinforcement learning

    Sutton, Richard S and Barto, Andrew G. Introduction to reinforcement learning. MIT Press, 1998

    work page 1998

  20. [20]

    Policy gradient methods for reinforcement learning with function approximation

    Sutton, Richard S, McAllester, David A, Singh, Satinder P, and Mansour, Yishay. Policy gradient methods for reinforcement learning with function approximation. In NIPS, volume 99, pp.\ 1057--1063. Citeseer, 1999

    work page 1999

  21. [21]

    Bias in natural actor-critic algorithms

    Thomas, Philip. Bias in natural actor-critic algorithms. In Proceedings of The 31st International Conference on Machine Learning, pp.\ 441--448, 2014

    work page 2014

  22. [22]

    Mujoco: A physics engine for model-based control

    Todorov, Emanuel, Erez, Tom, and Tassa, Yuval. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pp.\ 5026--5033. IEEE, 2012

    work page 2012

  23. [23]

    Real-time reinforcement learning by sequential actor--critics and experience replay

    Wawrzy \'n ski, Pawe . Real-time reinforcement learning by sequential actor--critics and experience replay. Neural Networks, 22 0 (10): 0 1484--1497, 2009

    work page 2009

  24. [24]

    Simple statistical gradient-following algorithms for connectionist reinforcement learning

    Williams, Ronald J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8 0 (3-4): 0 229--256, 1992

    work page 1992

  25. [25]

    Numerical optimization

    Wright, Stephen J and Nocedal, Jorge. Numerical optimization. Springer New York, 1999

    work page 1999