pith. sign in

arxiv: 2407.17032 · v4 · submitted 2024-07-24 · 💻 cs.LG · cs.DL

Gymnasium: A Standard Interface for Reinforcement Learning Environments

Mark Towers , Ariel Kwiatkowski , Jordan Terry , John U. Balis , Gianluca De Cola , Tristan Deleu , Manuel Goul\~ao , Andreas Kallinteris
show 8 more authors
Markus Krimmel Arjun KG Rodrigo Perez-Vicente Andrea Pierr\'e Sander Schulhoff Jun Jet Tai Hannah Tan Omar G. Younis
This is my paper

Pith reviewed 2026-05-11 17:24 UTC · model grok-4.3

classification 💻 cs.LG cs.DL
keywords reinforcement learningenvironment interfacestandard APIinteroperabilityreproducibilityopen source libraryRL environments
0
0 comments X

The pith

Gymnasium supplies a standard API for reinforcement learning environments to enable interoperability across implementations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Gymnasium as an open-source library that defines a common interface for reinforcement learning environments. It targets the fragmentation where incompatible environment and algorithm code makes it hard to compare results or extend prior work. The library supplies abstractions for compatibility, ready environments, customization tools, and reproducibility features. If the approach works, researchers spend less time on setup and more on testing new ideas. A reader would care because this kind of standardization has sped progress in other fields by letting people combine components without rewriting interfaces.

Core claim

Gymnasium is an open-source library that provides a standard API for RL environments. Its main feature is a set of abstractions that allow for wide interoperability between environments and training algorithms, making it easier for researchers to develop and test RL algorithms. In addition, Gymnasium provides a collection of easy-to-use environments, tools for easily customizing environments, and tools to ensure the reproducibility and robustness of RL research.

What carries the argument

The standard API consisting of abstractions that support interoperability between any compatible environment and any training algorithm.

If this is right

  • Different research groups can run each other's environments and algorithms without rewriting code.
  • Reproducibility checks become easier because the same environment definition works across labs.
  • New algorithms can be tested on a wider set of environments with minimal additional effort.
  • Customization tools allow rapid creation of variants while keeping the core interface intact.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Widespread use could turn the current collection of environments into a shared benchmark suite that new methods must beat.
  • The same abstraction pattern might be copied for simulation interfaces in robotics or game AI beyond reinforcement learning.
  • If adoption grows, older non-standard code bases may be wrapped once and then reused indefinitely under the new interface.

Load-bearing premise

That the reinforcement learning community will adopt this API broadly enough for the interoperability and reproducibility benefits to appear.

What would settle it

A count of recent RL papers that shows most new experiments still use custom or non-Gymnasium environment wrappers instead of the standard interface.

read the original abstract

Reinforcement Learning (RL) is a continuously growing field that has the potential to revolutionize many areas of artificial intelligence. However, despite its promise, RL research is often hindered by the lack of standardization in environment and algorithm implementations. This makes it difficult for researchers to compare and build upon each other's work, slowing down progress in the field. Gymnasium is an open-source library that provides a standard API for RL environments, aiming to tackle this issue. Gymnasium's main feature is a set of abstractions that allow for wide interoperability between environments and training algorithms, making it easier for researchers to develop and test RL algorithms. In addition, Gymnasium provides a collection of easy-to-use environments, tools for easily customizing environments, and tools to ensure the reproducibility and robustness of RL research. Through this unified framework, Gymnasium significantly streamlines the process of developing and testing RL algorithms, enabling researchers to focus more on innovation and less on implementation details. By providing a standardized platform for RL research, Gymnasium helps to drive forward the field of reinforcement learning and unlock its full potential. Gymnasium is available online at https://github.com/Farama-Foundation/Gymnasium

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 2 minor

Summary. The paper introduces Gymnasium, an open-source library that provides a standard API for reinforcement learning environments. Its core features are abstractions enabling interoperability between environments and training algorithms, a collection of environments, tools for environment customization, and tools to support reproducibility and robustness in RL research. The library aims to reduce implementation overhead and allow researchers to focus on innovation.

Significance. If the described API and tools see adoption, this is a significant practical contribution to the RL community by extending the standardization efforts of its predecessor (OpenAI Gym) with updated abstractions and reproducibility features. The open-source release with accompanying code and tools for robustness represents a concrete strength that can facilitate more comparable and reliable experiments across different environments and algorithms.

minor comments (2)
  1. The abstract repeats the theme of streamlining research and reducing implementation details in consecutive sentences; condensing this would improve conciseness and readability.
  2. The GitHub link is provided at the end of the abstract, but the manuscript would benefit from a brief dedicated paragraph or subsection early in the text describing installation, basic usage, and where to find documentation or example scripts.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive assessment of our manuscript and their recommendation to accept. We appreciate the recognition that Gymnasium extends the standardization efforts of OpenAI Gym with updated abstractions and reproducibility features, providing a practical contribution to the RL community.

Circularity Check

0 steps flagged

No circularity: purely descriptive software release with no derivations or predictions

full rationale

The manuscript is a software library announcement describing Gymnasium's API, environments, and reproducibility tools. It contains no equations, no fitted parameters, no predictions, and no derivation chain. The central claim is that the provided abstractions enable interoperability; this is presented as a design choice and implementation fact, not derived from prior results or self-citations. No load-bearing steps reduce to inputs by construction, and the paper does not invoke uniqueness theorems or ansatzes from prior work. The contribution is self-contained as documentation of an open-source interface.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is a software library announcement paper. It introduces no free parameters, mathematical axioms, or new postulated entities.

pith-pipeline@v0.9.0 · 5566 in / 1006 out tokens · 48714 ms · 2026-05-11T17:24:24.223074+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Efficient AI-Inspired Reduction of Feynman Integrals via Tube Seeding

    hep-ph 2026-06 unverdicted novelty 8.0

    Machine learning discovers a tube-seeding strategy for IBP reduction of Feynman integrals that scales linearly with numerator power, demonstrated on rank-20 2-loop 5-point integrals.

  2. Training Non-Differentiable Networks via Optimal Transport

    cs.LG 2026-05 unverdicted novelty 8.0

    PolyStep optimizes non-differentiable networks via forward-only polytope evaluations and optimal-transport barycentric updates, reaching 93.4% accuracy on hard-LIF spiking networks while outperforming gradient-free baselines.

  3. Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps

    cs.CR 2026-04 conditional novelty 8.0

    A new benchmark shows frontier LLMs achieve only 3.8% average recall identifying malicious events from raw logs and fail to meet 50% recall thresholds on most tactics.

  4. Leveraging Analytic Gradients in Provably Safe Reinforcement Learning

    cs.LG 2025-06 unverdicted novelty 8.0

    Develops and tests the first effective safeguard for analytic gradient-based provably safe RL, showing safe training on three control tasks without performance loss.

  5. STEMGym: Benchmarking Sequential Decision-Making under Dose Budgets in Autonomous Electron Microscopy

    cs.LG 2026-06 unverdicted novelty 7.0

    STEMGym benchmark demonstrates that perception pipelines dominate dose efficiency in autonomous STEM over navigation methods across 33 agent setups.

  6. Implicit Neural Representations of Individual Behavior

    cs.LG 2026-06 unverdicted novelty 7.0

    Behavioral INR adapts INRs to behavior by mapping states to actions with FiLM-modulated episode latents for self-supervised policy inference in unlabeled data, with new policy OOD definitions.

  7. Targeting World Models to Compromise Robot Learning Pipelines

    cs.RO 2026-06 unverdicted novelty 7.0

    World models introduce a stealthy poisoning vector into robot learning pipelines where malicious prompts or dynamics in teleoperated data activate only during synthetic trajectory generation, enabling backdoors in dow...

  8. Explainably Safe Reinforcement Learning

    cs.LG 2026-06 unverdicted novelty 7.0

    A hierarchical decision tree representation for non-deterministic safety shields in RL that classifies states by risk at design time and generates localized action explanations at runtime, yielding trees orders of mag...

  9. TuniQ: Autotuning Compilation Passes for Quantum Workloads at Scale for Effectiveness and Efficiency

    quant-ph 2026-05 unverdicted novelty 7.0

    TuniQ uses RL with a dual-encoder, shaped rewards, and action masking to autotune quantum compilation passes, improving fidelity and speed over Qiskit while generalizing across backends and scaling to large circuits.

  10. Learning When to Act: Communication-Efficient Reinforcement Learning via Run-Time Assurance

    cs.LG 2026-05 unverdicted novelty 7.0

    Learned policies with runtime Lyapunov shields achieve substantially higher communication intervals than baselines while maintaining stability on inverted pendulum, cart-pole, and quadrotor systems.

  11. Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shift

    cs.LG 2026-05 unverdicted novelty 7.0

    SeqRejectron constructs a stopping rule with a small set of validator policies to achieve horizon-free sample complexity for selective imitation learning under arbitrary dynamics shifts.

  12. Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shift

    cs.LG 2026-05 unverdicted novelty 7.0

    SeqRejectron builds a stopping rule from a small set of validator policies to achieve horizon-free sample-complexity guarantees for selective imitation learning under arbitrary train-test dynamics shifts.

  13. Agentick: A Unified Benchmark for General Sequential Decision-Making Agents

    cs.AI 2026-05 unverdicted novelty 7.0

    Agentick is a new benchmark for sequential decision-making agents that evaluates RL, LLM, VLM, hybrid, and human approaches across 37 tasks and finds no single method dominates.

  14. Beyond the Independence Assumption: Finite-Sample Guarantees for Deep Q-Learning under $\tau$-Mixing

    stat.ML 2026-05 unverdicted novelty 7.0

    Finite-sample risk bounds for DQN with ReLU networks are extended to τ-mixing data, showing an extra dimensionality penalty in the convergence rate due to dependence.

  15. SOPE: Stabilizing Off-Policy Evaluation for Online RL with Prior Data

    cs.LG 2026-05 unverdicted novelty 7.0

    SOPE uses an actor-aligned OPE signal on a held-out validation split to dynamically stop offline stabilization phases in online RL, improving performance up to 45.6% and cutting TFLOPs up to 22x on 25 Minari tasks.

  16. Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior

    cs.LG 2026-05 unverdicted novelty 7.0

    Manifold steering along activation geometry induces behavioral trajectories matching the natural manifold of outputs, while linear steering produces off-manifold unnatural behaviors.

  17. Stable GFlowNets with Probabilistic Guarantees

    cs.LG 2026-05 unverdicted novelty 7.0

    Derives loss-to-TV bounds providing probabilistic guarantees for GFlowNets and introduces Stable GFlowNets algorithm for improved training stability and distributional fidelity.

  18. PACE: Parameter Change for Unsupervised Environment Design

    cs.LG 2026-05 unverdicted novelty 7.0

    PACE uses the squared L2 norm of policy parameter changes from a first-order approximation as an efficient proxy for environment value in UED, outperforming baselines with higher IQM and lower optimality gap on MiniGr...

  19. A Multi-View Media Profiling Suite: Resources, Evaluation, and Analysis

    cs.CL 2026-05 unverdicted novelty 7.0

    Presents MBFC-2025 dataset and multi-view embeddings with fusion methods for media bias and factuality, reporting SOTA results on ACL-2020 and new benchmarks on MBFC-2025.

  20. Your Loss is My Gain: Low Stake Attacks on Liquid Staking Pools

    cs.GT 2026-05 unverdicted novelty 7.0

    A low-stake adversary can degrade a liquid staking pool's performance via consensus manipulation and profit from the resulting drop in its LST value through application-layer financial positions.

  21. KinDER: A Physical Reasoning Benchmark for Robot Learning and Planning

    cs.RO 2026-04 unverdicted novelty 7.0

    KinDER is a new open-source benchmark that demonstrates substantial gaps in current robot learning and planning methods for handling physical constraints.

  22. SpecRLBench: A Benchmark for Generalization in Specification-Guided Reinforcement Learning

    cs.LG 2026-04 unverdicted novelty 7.0

    SpecRLBench is a new benchmark evaluating generalization of LTL-guided RL methods across navigation and manipulation domains with static/dynamic environments and varied robot dynamics.

  23. Replay-buffer engineering for noise-robust quantum circuit optimization

    quant-ph 2026-04 unverdicted novelty 7.0

    Treating the replay buffer as a central lever in RL for quantum circuit optimization yields 4-32x sample efficiency gains, up to 67.5% faster episodes, and 85-90% fewer steps to accuracy on noisy molecular and compila...

  24. Beyond Bellman: High-Order Generator Regression for Continuous-Time Policy Evaluation

    stat.ML 2026-04 unverdicted novelty 7.0

    High-order generator regression from multi-step trajectories yields a second-order accurate estimator for finite-horizon continuous-time policy evaluation that outperforms the Bellman baseline in calibration studies a...

  25. GPU-Accelerated Continuous-Time Successive Convexification for Contact-Implicit Legged Locomotion

    cs.RO 2026-04 unverdicted novelty 7.0

    ci-SCvx introduces integral cross-complementarity constraints to continuous-time SCP for contact-implicit legged locomotion, achieving faster solves and lower energy in MuJoCo validation than MPC baselines.

  26. SafeAdapt: Provably Safe Policy Updates in Deep Reinforcement Learning

    cs.LG 2026-04 unverdicted novelty 7.0

    SafeAdapt certifies a Rashomon set of safe policies from demonstration data and projects updates from arbitrary RL algorithms onto it to guarantee preservation of safety on source tasks.

  27. Automatic Generation of High-Performance RL Environments

    cs.LG 2026-03 conditional novelty 7.0

    Closed-loop prompt-based translation with hierarchical verification and iterative repair produces equivalent high-performance RL environments across five cases including new TCGJax.

  28. RLGT: A reinforcement learning framework for extremal graph theory

    cs.LG 2026-02 unverdicted novelty 7.0

    RLGT is a modular reinforcement learning framework for extremal graph theory that handles undirected, directed, looped, and multi-colored graphs to facilitate future research.

  29. About Time: Model-free Reinforcement Learning with Timed Reward Machines

    cs.AI 2025-12 conditional novelty 7.0

    Timed reward machines extend reward machines with timing constraints, allowing model-free RL algorithms to learn policies that satisfy precise temporal requirements on standard benchmarks.

  30. Sample-Efficient and Smooth Cross-Entropy Method Model Predictive Control Using Deterministic Samples

    eess.SY 2025-10 unverdicted novelty 7.0

    dsCEM replaces random sampling in CEM-MPC with deterministic samples from localized cumulative distributions to improve efficiency and smoothness in nonlinear optimal control.

  31. Frictional Q-Learning

    cs.LG 2025-09 unverdicted novelty 7.0

    Frictional Q-Learning encodes supported actions as tangent directions on an action manifold using a contrastive variational autoencoder to reduce extrapolation errors in off-policy reinforcement learning.

  32. Adaptive Ensemble Aggregation for Actor-Critics

    cs.LG 2025-07 unverdicted novelty 7.0

    AEA dynamically aggregates ensembles in off-policy actor-critics from training dynamics, with proofs of convergence to an error-minimizing equilibrium, bias shrinkage with ensemble size, and monotonic policy improvement.

  33. DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertainty

    cs.LG 2025-06 unverdicted novelty 7.0

    DR-SAC is the first actor-critic distributionally robust RL algorithm for offline continuous control that derives a convergent robust soft policy iteration and reports up to 9.8x higher rewards than SAC under perturbations.

  34. Parameter-Efficient Distributional RL via Normalizing Flows and a Geometry-Aware Cram\'er Surrogate

    cs.AI 2025-05 unverdicted novelty 7.0

    NFDRL models return distributions via continuous normalizing flows paired with a geometry-aware Cramér surrogate distance, delivering fixed-size parameters, a sqrt(gamma) contraction, unbiased gradients, and competiti...

  35. Simultaneous Multi-die Floorplanning and Technology Assignment

    eess.SY 2025-02 unverdicted novelty 7.0

    A joint optimization framework for multi-die floorplanning and technology assignment that uses ML-based PPA estimation to optimize area, wirelength, performance, power, and cost, outperforming greedy baselines in 2.5D...

  36. EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments

    cs.AI 2026-07 unverdicted novelty 6.0

    EvoPolicyGym is a new benchmark suite of 16 compact RL environments that evaluates autonomous policy evolution, with GPT-5.5 achieving the top aggregate rank and top-two performance on all tasks.

  37. From Consistency to Collaborative Discovery: MFEA-CoD for Multitask Novelty Search

    cs.NE 2026-07 unverdicted novelty 6.0

    MFEA-CoD coordinates novelty search tasks with a multitask repulsion operator and adaptive inter-task transfer to discover diverse novel solutions more efficiently across synthetic, maze, MuJoCo, and generative problems.

  38. From Consistency to Collaborative Discovery: MFEA-CoD for Multitask Novelty Search

    cs.NE 2026-07 unverdicted novelty 6.0

    MFEA-CoD coordinates novelty search tasks with repulsion and adaptive transfer to collaboratively discover diverse novel solutions across synthetic, maze, MuJoCo, and generative problems.

  39. Learning Expert Strategy for Autonomous Robotic Endovascular Intervention via Decoupled Procedural Execution

    cs.RO 2026-06 unverdicted novelty 6.0

    Decoupled strategic RL policy and expert-informed execution module for autonomous endovascular navigation, reporting >96% success rate, 29.3% fewer steps, and 13% less trajectory variance in simulation and real robot tests.

  40. Failure-Based Testing for Deep Reinforcement Learning Agents

    cs.SE 2026-06 unverdicted novelty 6.0

    Proposes Prior Random Testing (PRT) that leverages task difficulty to prioritize failure-prone test cases for DRL agents, achieving over 50% lower testing cost than random testing while preserving diversity on four be...

  41. RoAd-RL: A Unified Library and Benchmark for Robust Adversarial Reinforcement Learning

    cs.LG 2026-06 conditional novelty 6.0

    RoAd-RL is a new benchmarking library for adversarial reinforcement learning that evaluates DQN, PPO, and SAC agents across 192 attack-defense configurations and finds substantial robustness variations plus cases wher...

  42. PAWS: Preference Learning with Advantage-Weighted Segments

    cs.LG 2026-06 unverdicted novelty 6.0

    PAWS performs policy updates directly using segment-level advantage functions to align utility training with policy optimization in preference-based RL.

  43. MuJoCo-Drones-Gym: A GPU-Accelerated Multi-Drone Simulator for Control and Reinforcement Learning

    cs.RO 2026-06 unverdicted novelty 6.0

    Introduces MuJoCo-Drones-Gym, a modular Gymnasium-compatible multi-drone simulator on MuJoCo with GPU acceleration and seven example tasks for control and RL.

  44. MineXplore: An Open-Source Reinforcement Learning Exploration Benchmark for GNSS-Denied Underground Environment

    cs.RO 2026-06 unverdicted novelty 6.0

    MineXplore provides a MuJoCo-based underground mine navigation benchmark derived from real data, with geometric validation (IoU 0.9538) and a PPO baseline reaching 88.89% coverage.

  45. Adversarial Dual On-Policy Distillation from Expressive Teacher

    cs.LG 2026-05 unverdicted novelty 6.0

    FA-OPD co-trains a flow-matching teacher and MLP student via adversarial dual on-policy distillation, improving robustness over baselines on six robot benchmarks with noisy or limited demonstrations.

  46. Cost-Aware Adaptive Conformal Inference for Runtime Assurance in Dynamic Environments

    eess.SY 2026-05 unverdicted novelty 6.0

    Proposes Cost-Aware Adaptive Conformal Inference framework providing dual statistical guarantees on long-run violation frequency and cumulative violation cost for runtime assurance in dynamic environments.

  47. Score-Based One-step MeanFlow Policy Optimization

    cs.LG 2026-05 unverdicted novelty 6.0

    SOM is an actor-critic algorithm that constructs the target velocity field for one-step MeanFlow policies directly from the Q-function via score estimation and probability flow ODE, achieving claimed SOTA on locomotio...

  48. Hack-Verifiable Environments: Towards Evaluating Reward Hacking at Scale

    cs.LG 2026-05 unverdicted novelty 6.0

    Presents Hack-Verifiable TextArena, a benchmark that embeds verifiable reward hacking opportunities into environments to enable deterministic measurement of exploitation by language models.

  49. Critic-Driven Voronoi-Quantization for Distilling Deep RL Policies to Explainable Models

    cs.LG 2026-05 unverdicted novelty 6.0

    Critic-Driven Voronoi State Partitioning distills deep RL policies into piecewise-linear models by iteratively adding linear subpolicies in high-value-error regions identified by the critic.

  50. Learning Selective Merge Policies for Deadline-Constrained Coded Caching via Deep Reinforcement Learning

    cs.IT 2026-05 unverdicted novelty 6.0

    A DRL policy with graph attention learns selective merging for deadline-constrained coded caching, cutting packet expiration ratio by 40.9% versus SACM++ while merging only about 32% of the time.

  51. Augmented Lagrangian Method for Last-Iterate Convergence for Constrained MDPs

    cs.LG 2026-05 unverdicted novelty 6.0

    An inexact augmented Lagrangian method with projected Q-ascent yields global last-iterate convergence guarantees for constrained MDP policy optimization, extending from tabular to log-linear and non-linear policies.

  52. Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias

    cs.LG 2026-05 unverdicted novelty 6.0

    Reflective Prompted Policy Optimization uses a Critic-LLM to inspect full trajectories and propose grounded revisions, yielding higher mean best rewards, faster near-optimal performance, and greater stability than sca...

  53. Agentick: A Unified Benchmark for General Sequential Decision-Making Agents

    cs.AI 2026-05 unverdicted novelty 6.0

    Agentick is a new unified benchmark for sequential decision-making agents that evaluates RL, LLM, VLM, hybrid, and human agents across 37 tasks, showing no single approach dominates.

  54. SOPE: Stabilizing Off-Policy Evaluation for Online RL with Prior Data

    cs.LG 2026-05 conditional novelty 6.0

    SOPE dynamically controls offline training length in online RL using actor-aligned OPE on validation data to stop when benefits saturate, achieving up to 45.6% better performance and 22x less computation on Minari tasks.

  55. Extending Differential Temporal Difference Methods for Episodic Problems

    cs.LG 2026-05 unverdicted novelty 6.0

    A generalization of differential TD extends it to episodic settings while preserving policy ordering, inheriting linear TD guarantees, and improving sample efficiency.

  56. ANO: A Principled Approach to Robust Policy Optimization

    cs.AI 2026-05 unverdicted novelty 6.0

    ANO derives a robust policy optimizer from geometric principles that replaces clipping with a smooth redescending gradient, showing better performance and stability than PPO, SPO, and GRPO in MuJoCo, Atari, and RLHF e...

  57. Bridging the Gap Between Average and Discounted TD Learning

    cs.LG 2026-05 unverdicted novelty 6.0

    A new two-trajectory sampling algorithm for average-reward TD learning guarantees convergence with quadratic sample complexity and no explicit dimension dependence in both tabular and linear approximation settings.

  58. Perturb and Correct: Post-Hoc Ensembles using Affine Redundancy

    cs.LG 2026-05 unverdicted novelty 6.0

    Perturb-and-Correct generates epistemically diverse predictors from a single pretrained network via hidden-layer perturbations followed by affine least-squares corrections that enforce agreement on calibration data.

  59. Breaking the Computational Barrier: Provably Efficient Actor-Critic for Low-Rank MDPs

    cs.LG 2026-05 unverdicted novelty 6.0

    An actor-critic RL algorithm for low-rank MDPs achieves improved sample efficiency using solely a policy evaluation oracle.

  60. Beyond Bellman: High-Order Generator Regression for Continuous-Time Policy Evaluation

    stat.ML 2026-04 unverdicted novelty 6.0

    High-order moment-matching estimation of the time-dependent generator improves continuous-time policy evaluation accuracy over first-order Bellman recursion by canceling lower-order truncation terms, with supporting e...

Reference graph

Works this paper leans on

34 extracted references · 34 canonical work pages · cited by 107 Pith papers · 5 internal anchors

  1. [1]

    Hindsight Experience Replay

    URL https://arxiv.org/abs/1707.01495. Charles Beattie, Thomas Köppe, Edgar A. Duéñez-Guzmán, and Joel Z. Leibo. Deepmind lab2d,

    work page Pith review arXiv

  2. [2]

    URL https://arxiv.org/abs/2011.07027. M. G. Bellemare, Y . Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, jun

    work page arXiv 2011

  3. [3]

    Dota 2 with Large Scale Deep Reinforcement Learning

    Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680,

    work page internal anchor Pith review arXiv 1912

  4. [4]

    Akhilan Boopathy, Aneesh Muppidi, Peggy Yang, Abhiram Iyer, William Yue, and Ila Fiete

    URL https://arxiv.org/abs/2306.09884. Felix Book, Arne Traue, Maximilian Schenke, Barnabas Haucke-Korber, and Oliver Wallscheid. Gym-electric-motor (gem) control: An automated open-source controller design suite for drives. In 2023 IEEE International Electric Machines & Drives Conference (IEMDC), pages 1–7,

    work page arXiv 2023

  5. [5]

    10 Yann Bouteiller, Edouard GEZE, GobeX, Stefan Kuhn, and pius

    doi: 10.1109/IEMDC55163.2023.10239044. 10 Yann Bouteiller, Edouard GEZE, GobeX, Stefan Kuhn, and pius. trackmania-rl/tmrl: Release 0.7.1, May

    work page doi:10.1109/iemdc55163.2023.10239044 2023

  6. [6]

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang

    URL https://doi.org/10.5281/zenodo.15344778. James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs,

    work page doi:10.5281/zenodo.15344778

  7. [7]

    OpenAI Gym

    URL http://github.com/google/jax. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540,

    work page internal anchor Pith review arXiv

  8. [8]

    Dopamine: A Research Framework for Deep Reinforcement Learning

    URL http: //arxiv.org/abs/1812.06110. Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo Perez-Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neural Information Processing Systems 36, New...

    work page Pith review arXiv

  9. [9]

    arXiv preprint arXiv:2406.05646 , year=

    URL https://arxiv.org/ abs/2406.05646. Steven Dalton, Iuri Frosio, and Michael Garland. Accelerating reinforcement learning through gpu atari emulation,

    work page arXiv

  10. [10]

    DeepMind

    URL https://arxiv.org/abs/1907.08467. Rodrigo de Lazcano, Kallinteris Andreas, Jun Jet Tai, Seungjae Ryan Lee, and Jordan Terry. Gymnasium robotics,

    work page arXiv 1907

  11. [11]

    arXiv:2306.08649

    URL https: //arxiv.org/abs/2306.08649. Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https: //github.com/openai/baselines,

    work page arXiv

  12. [12]

    Alegre, Ann Nowé, Ana L

    Florian Felten, Lucas N. Alegre, Ann Nowé, Ana L. C. Bazzan, El Ghazali Talbi, Grégoire Danoy, and Bruno C. da Silva. A toolkit for reliable benchmarking and research in multi-objective reinforcement learning. In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023),

    work page 2023

  13. [14]

    Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

    URL http://arxiv.org/abs/1801.01290. arXiv: 1801.01290 version:

    work page internal anchor Pith review arXiv

  14. [16]

    Acme: A research framework for distributed reinforcement learning

    URL https://arxiv.org/abs/2006.00979. Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep rein- forcement learning algorithms. Journal of Machine Learning Research, 23(274):1–18,

    work page arXiv 2006

  15. [18]

    Hyde, G.; and Santos Jr, E

    URL https://arxiv.org/abs/2402.03046. Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1):99–134, May

    work page arXiv

  16. [19]

    Littman and Anthony R

    ISSN 0004-3702. doi: 10.1016/S0004-3702(98)00023-X. Saman Kazemkhani, Aarav Pandya, Daphne Cornelisse, Brennan Shacklett, and Eugene Vinitsky. Gpudrive: Data-driven, multi-agent driving simulation at 1 million fps. In Proceedings of the International Conference on Learning Representations (ICLR) ,

    work page doi:10.1016/s0004-3702(98)00023-x

  17. [20]

    Gpudrive: Data-driven, multi-agent driving simulation at 1 million fps,

    URL https://arxiv. org/abs/2408.01584. Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Ja´skowski. ViZ- Doom: A Doom-based AI research platform for visual reinforcement learning. InIEEE Conference on Computational Intelligence and Games, pages 341–348, Santorini, Greece, Sep

    work page arXiv

  18. [21]

    ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning

    IEEE. URL http://arxiv.org/abs/1605.02097. The best paper award. Sotetsu Koyamada, Shinri Okano, Soichiro Nishimori, Yu Murata, Keigo Habara, Haruka Kita, and Shin Ishii. Pgx: Hardware-accelerated parallel game simulators for reinforcement learning. In Advances in Neural Information Processing Systems,

    work page Pith review arXiv

  19. [22]

    Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration

    URL https://arxiv.org/abs/1802.08802. Mikel Malagón, Josu Ceberio, and Jose A. Lozano. Craftium: An extensible framework for creating reinforcement learning environments. arXiv preprint arXiv:2407.03969,

    work page Pith review arXiv

  20. [23]

    Playing Atari with Deep Reinforcement Learning

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602,

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [24]

    Rusu, Joel Veness, Marc G

    ISSN 1476-4687. doi: 10.1038/nature14236. URL https://www.nature.com/articles/nature14236. Alistair Muldal, Yotam Doron, John Aslanides, Tim Harley, Tom Ward, and Siqi Liu. dm_env: A python interface for reinforcement learning environments,

    work page doi:10.1038/nature14236

  22. [25]

    D., Conway, A., Cowan, N., Donkin, C., Farrell, S., Hitch, G

    URL https://arxiv.org/abs/1908.03568. Mathieu Poliquin. Stable retro, a maintained fork of openai’s gym-retro. https://github.com/ Farama-Foundation/stable-retro,

    work page arXiv 1908

  23. [26]

    Proximal Policy Optimization Algorithms

    URL http://jmlr.org/papers/v22/20-1364.html. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs], August

    work page internal anchor Pith review Pith/arXiv arXiv

  24. [27]

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al

    URL https://arxiv.org/ abs/2203.16777. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354–359,

    work page arXiv

  25. [28]

    Pufferlib: Making reinforcement learning libraries and environments play nice, 2024

    URL https://arxiv.org/abs/2406.12905. Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 12th International Conference on Neural Information Processing Systems, NIPS’99, pages 1057–1063, Cambridge, MA, USA, November

    work page arXiv

  26. [29]

    Jun Jet Tai, Jim Wong, Mauro Innocente, Nadjim Horri, James Brusey, and Swee King Phang

    MIT Press. Jun Jet Tai, Jim Wong, Mauro Innocente, Nadjim Horri, James Brusey, and Swee King Phang. Pyflyt– uav simulation environments for reinforcement learning research.arXiv preprint arXiv:2304.01305,

    work page arXiv

  27. [30]

    Pettingzoo: Gym for multi-agent reinforcement learning

    J Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Perez-Vicente, et al. Pettingzoo: Gym for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34: 15032–15043, 2021a. J. K. Terry, Benjamin Black, and Luis Santos. Multipla...

    work page arXiv 2009

  28. [31]

    In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems

    doi: 10.1109/IROS.2012.6386109. Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6:100022,

    work page doi:10.1109/iros.2012.6386109 2012

  29. [32]

    Mengjie Chen, Chao Gao, and Zhao Ren

    ISSN 2665-9638. doi: https://doi.org/ 10.1016/j.simpa.2020.100022. URL https://www.sciencedirect.com/science/article/ pii/S2665963820300099. 13 Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi...

    work page doi:10.1016/j.simpa.2020.100022 2020

  30. [34]

    arXiv: 2107.14171

    URL http://arxiv.org/abs/2107.14171. arXiv: 2107.14171. Jiayi Weng, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu, Yufan Song, Ting Luo, Yukun Jiang, Zhongwen Xu, and Shuicheng Yan. En- vPool: A highly parallel reinforcement learning environment execution engine. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho...

    work page arXiv

  31. [35]

    Kenny Young and Tian Tian

    URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 8caaf08e49ddbad6694fae067442ee21-Paper-Datasets_and_Benchmarks.pdf. Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments,

    work page 2022

  32. [36]

    URL https://arxiv.org/abs/1903.03176. Omar G. Younis, Rodrigo Perez-Vicente, John U. Balis, Will Dudley, Alex Davey, and Jordan K Terry. Minari, September

    work page Pith review arXiv 1903

  33. [37]

    Siyuan Zhang and Nan Jiang

    URL https://doi.org/10.5281/zenodo.13767625. Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL),

    work page doi:10.5281/zenodo.13767625

  34. [38]

    URL https://arxiv.org/abs/1910.10897. 14

    work page arXiv 1910