Reinforcement and Imitation Learning via Interactive No-Regret Learning
read the original abstract
Recent work has demonstrated that problems-- particularly imitation learning and structured prediction-- where a learner's predictions influence the input-distribution it is tested on can be naturally addressed by an interactive approach and analyzed using no-regret online learning. These approaches to imitation learning, however, neither require nor benefit from information about the cost of actions. We extend existing results in two directions: first, we develop an interactive imitation learning approach that leverages cost information; second, we extend the technique to address reinforcement learning. The results provide theoretical support to the commonly observed successes of online approximate policy iteration. Our approach suggests a broad new family of algorithms and provides a unifying view of existing techniques for imitation and reinforcement learning.
This paper has not been read by Pith yet.
Forward citations
Cited by 12 Pith papers
-
Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback
Noisy expert imitation learning requires exponential samples for offline methods but polynomial for a variant of on-policy distillation under a noise condition.
-
When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizon
Online IL overcomes an information-theoretic bottleneck that offline IL faces in non-realizable settings even at horizon 1, under a new structural characterization of reward-relative misspecification.
-
Speculative Rollback Correction for Quality-Diverse Web Agent Imitation
SRC is a fixed-horizon branch review framework for imitation learning in resettable web environments that collects 977 verifier-passing trajectories and 9,183 next-action examples while improving recovery-versus-query...
-
Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approach
Introduces a feasible-reward-set approach to IRL with multiple heterogeneous suboptimal demonstrators, proving monotonic shrinkage of the joint set and two recovery guarantees for the ground-truth optimal reward.
-
Revisiting DAgger in the Era of LLM-Agents
DAgger-style training with turn-level policy interpolation raises 4B and 8B LLM agents to 27.3% and 29.8% on SWE-bench Verified, beating several larger published systems.
-
Provable imitation learning for control of instability in partially-observed Vlasov--Poisson equations
Imitation learning yields provably stable feedback policies for partially observed Vlasov-Poisson instabilities, with error floors determined by minimal behavior-cloning loss characterized via entropy of the initial d...
-
BridgeSim: Unveiling the OL-CL Gap in End-to-End Autonomous Driving
The primary OL-CL gap in end-to-end autonomous driving arises from objective mismatch creating structural inability to model reactive behaviors, which a test-time adaptation method can mitigate.
-
Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning
Process advantage verifiers trained to predict step-level progress under a distinct prover policy improve LLM reasoning accuracy by over 8% and sample efficiency by 5-6x over outcome reward models.
-
Learning Belief Representations for Imitation Learning in POMDPs
BMIL learns belief modules jointly with policies for GAIL-style imitation learning in POMDPs, outperforming separate training and standard GAIL on continuous control tasks.
-
The Essence of Balance for Self-Improving Agents in Vision-and-Language Navigation
SDB balances behavioral diversity and learning stability in VLN self-improvement by expanding decisions into latent hypotheses, performing reliability-aware aggregation, and applying a regularizer, yielding gains such...
-
Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents
VLAJS augments PPO with sparse annealed VLA guidance through directional regularization to cut required interactions by over 50% on manipulation tasks and enable zero-shot sim-to-real transfer.
-
State-Conditional Adversarial Learning: An Off-Policy Visual Domain Transfer Method for End-to-End Imitation Learning
SCAL derives an upper bound on target-domain imitation loss using source loss plus state-conditional latent KL divergence and aligns distributions via a discriminator-based adversarial estimator.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.