REVIEW 4 cited by
GenDICE: Generalized Offline Estimation of Stationary Values
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
GenDICE: Generalized Offline Estimation of Stationary Values
read the original abstract
An important problem that arises in reinforcement learning and Monte Carlo methods is estimating quantities defined by the stationary distribution of a Markov chain. In many real-world applications, access to the underlying transition operator is limited to a fixed set of data that has already been collected, without additional interaction with the environment being available. We show that consistent estimation remains possible in this challenging scenario, and that effective estimation can still be achieved in important applications. Our approach is based on estimating a ratio that corrects for the discrepancy between the stationary and empirical distributions, derived from fundamental properties of the stationary distribution, and exploiting constraint reformulations based on variational divergence minimization. The resulting algorithm, GenDICE, is straightforward and effective. We prove its consistency under general conditions, provide an error analysis, and demonstrate strong empirical performance on benchmark problems, including off-line PageRank and off-policy policy evaluation.
Forward citations
Cited by 4 Pith papers
-
Fitted $Q$ Evaluation Without Bellman Completeness via Stationary Weighting
Stationary-weighted FQE achieves finite-sample linear convergence to the projected Bellman fixed point without Bellman completeness by reweighting regressions to the target stationary norm.
-
Bellman Calibration for $V$-Learning in Offline Reinforcement Learning
Bellman calibration supplies a new reliability criterion and post-hoc recalibration method for value functions in offline RL, with finite-sample guarantees at one-dimensional nonparametric rates that avoid Bellman com...
-
Stationary Robust Mean-Field Games under Model Mismatches
Develops infinite-horizon stationary robust mean-field games incorporating distributional uncertainty, proves equilibrium existence via fixed-point on contractive Bellman operator, gives convergent algorithm, and deri...
-
Stationary Reweighting Yields Local Convergence of Soft Fitted Q-Iteration
Stationary reweighting of soft fitted Q-iteration yields finite-sample local linear convergence to the projected fixed point under approximate realizability and controlled weighting error, even without Bellman completeness.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.