pith. sign in

arxiv: 1906.09237 · v2 · pith:ZJXBSQP4new · submitted 2019-06-21 · 💻 cs.LG · cs.AI· stat.ML

Shaping Belief States with Generative Environment Models for RL

Karol Gregor , Danilo Jimenez Rezende , Frederic Besse , Yan Wu , Hamza Merzic , Aaron van den Oord This is my paper

Pith reviewed 2026-05-25 18:54 UTC · model grok-4.3

classification 💻 cs.LG cs.AIstat.ML
keywords generative modelsbelief statesreinforcement learningpredictive models3D environmentsdata efficiencyworld modelsovershooting
0
0 comments X

The pith

A generative model trained to predict future observations forms stable belief states that capture 3D environment layout and agent position.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows how to train expressive generative models efficiently so they can build and maintain belief states about complex environments. In visually rich dynamic 3D settings these states encode the overall layout together with the agent's position and orientation. When the resulting representations feed into reinforcement learning they produce clear gains in data efficiency over model-free baselines. The training relies on predicting multiple steps ahead, and the authors supply a practical method to keep the computational cost manageable.

Core claim

A predictive algorithm with an expressive generative model can form stable belief-states in visually rich and dynamic 3D environments. More precisely, the learned representation captures the layout of the environment as well as the position and orientation of the agent. Experiments show the model substantially improves data-efficiency on a number of reinforcement learning tasks compared with strong model-free baseline agents, with multi-step prediction proving critical for stable representations.

What carries the argument

Expressive generative model trained with multi-step future prediction (overshooting) to shape agent belief states.

If this is right

  • Multi-step prediction is required for the emergence of stable belief states.
  • Belief states that encode layout and pose produce measurable data-efficiency gains on RL tasks.
  • A dedicated reduction scheme keeps the approach competitive in wall-clock time despite generative-model cost.
  • The representation learns spatial structure implicitly through prediction alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same predictive training could be tested on tasks where agents must transfer across environments that share layout but differ in appearance.
  • Longer overshooting horizons might further improve pose and layout accuracy in larger or more dynamic spaces.
  • The method suggests generative prediction can substitute for explicit supervision when learning structured world representations.

Load-bearing premise

Data-efficiency gains on the RL tasks come from the quality of the learned belief states rather than from extra compute or different optimization schedules.

What would settle it

An ablation that matches the generative agent's total compute and training schedule exactly while removing the generative model, then checks whether the reported data-efficiency advantage disappears.

Figures

Figures reproduced from arXiv: 1906.09237 by Aaron van den Oord, Danilo Jimenez Rezende, Frederic Besse, Hamza Merzic, Karol Gregor, Yan Wu.

Figure 1
Figure 1. Figure 1: Diagram of the agent and model. The agent receives observations [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Random City environment. Rows: 1. Input to the model sequence starting from the beginning of the episode. 2. Top down view (a map). 3. Top down view decoded from the belief state. The belief state was not trained with this decoding signal, but only from the first person view (top row). We see that the model is able to fill up the map as it sees new frames. 4. Frames later in the sequence (after 170 steps).… view at source ↗
Figure 3
Figure 3. Figure 3: The choice of model and overshoot length have significant impact on state representation. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effect of overshoot on environment’s map decoding. This analysis shows that Generative [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Generative SimCore results in substantial data-efficiency gains for agents in DeepMind-Lab [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The input and the rollout in DeepMind Lab. The agent is able to correctly rollout for many [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Top: Voxel levels. There are four levels: BridgeFood, Cliff, Food and HighFood. For each [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 12
Figure 12. Figure 12: This environment is harder than the city, because it takes significantly more steps to cross [PITH_FULL_IMAGE:figures/full_fig_p008_12.png] view at source ↗
Figure 8
Figure 8. Figure 8: Diagram of ConvDraw’s likelihood computation. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Illustration of a dynamic Kanerva machine integrated with an RNN. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Effect of the choice of GECO’s κ threshold on map-reconstruction using SimCore. We find that a value κ ≈ 1e-3 produces the best results for map reconstruction. Appendix H Map decoding on Random City environment Here we show additional map decoding samples for each of model in [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Additional map decoding samples for each model. All models were trained for the same [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Terrain. The agent moves around a procedurally generated terrain. The first row in each of [PITH_FULL_IMAGE:figures/full_fig_p019_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Illustration of the Map decoding MSE along a single trajectory in the procedural terrain [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Inputs and samples from the building levels. First two rows show input and samples from [PITH_FULL_IMAGE:figures/full_fig_p021_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Inputs and samples from the terrain environment. [PITH_FULL_IMAGE:figures/full_fig_p022_15.png] view at source ↗
read the original abstract

When agents interact with a complex environment, they must form and maintain beliefs about the relevant aspects of that environment. We propose a way to efficiently train expressive generative models in complex environments. We show that a predictive algorithm with an expressive generative model can form stable belief-states in visually rich and dynamic 3D environments. More precisely, we show that the learned representation captures the layout of the environment as well as the position and orientation of the agent. Our experiments show that the model substantially improves data-efficiency on a number of reinforcement learning (RL) tasks compared with strong model-free baseline agents. We find that predicting multiple steps into the future (overshooting), in combination with an expressive generative model, is critical for stable representations to emerge. In practice, using expressive generative models in RL is computationally expensive and we propose a scheme to reduce this computational burden, allowing us to build agents that are competitive with model-free baselines.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The manuscript proposes training expressive generative models to form stable belief states for RL agents in visually rich, dynamic 3D environments. It claims that a predictive algorithm using such models, with multi-step future prediction (overshooting), yields representations that capture environment layout as well as agent position and orientation, leading to substantially improved data efficiency on RL tasks relative to strong model-free baselines; a computational reduction scheme is introduced to address the expense of expressive generative models.

Significance. If the reported gains can be shown to arise specifically from the quality of the learned belief states under controlled conditions, the work would offer a useful empirical demonstration of how generative models can shape representations for RL in complex visual settings and would strengthen the case for overshooting in predictive training.

major comments (2)
  1. [Abstract] Abstract: the claim that the model 'substantially improves data-efficiency on a number of reinforcement learning (RL) tasks' is stated without any quantitative results, ablation tables, error bars, or statistical tests, so the central empirical assertion cannot be assessed.
  2. [Abstract and setup] Abstract and setup: comparisons to model-free baselines do not report matched total FLOPs, wall-clock time, or optimizer schedules, leaving the performance delta unattributable to belief-state quality versus unablated differences in compute or training dynamics.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below and outline revisions to improve clarity and assessability of the empirical claims.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claim that the model 'substantially improves data-efficiency on a number of reinforcement learning (RL) tasks' is stated without any quantitative results, ablation tables, error bars, or statistical tests, so the central empirical assertion cannot be assessed.

    Authors: We agree the abstract would be stronger with concrete numbers. The body of the manuscript reports detailed results including ablation studies, learning curves with error bars across multiple seeds, and statistical comparisons on several RL tasks. In revision we will add specific quantitative highlights (e.g., relative sample-efficiency gains) to the abstract while preserving its length. revision: yes

  2. Referee: [Abstract and setup] Abstract and setup: comparisons to model-free baselines do not report matched total FLOPs, wall-clock time, or optimizer schedules, leaving the performance delta unattributable to belief-state quality versus unablated differences in compute or training dynamics.

    Authors: The model-free baselines follow the standard training protocols and hyper-parameters reported in their source papers (e.g., A2C, PPO variants) to ensure reproducibility. Our primary metric is data efficiency measured in environment steps; compute-matched experiments were not performed. We will add a new paragraph in the experimental section providing approximate FLOPs estimates for both model classes and a brief discussion of wall-clock differences, allowing readers to judge the trade-off. revision: partial

Circularity Check

0 steps flagged

No circularity: empirical demonstration with independent experimental support

full rationale

The paper advances an empirical claim that expressive generative models trained with multi-step prediction form stable belief states that improve RL data efficiency in 3D environments. No equations, uniqueness theorems, or first-principles derivations are presented that reduce the reported outcomes to quantities defined by the model's own fitted parameters or prior self-citations. The core results rest on experimental comparisons to model-free baselines and ablations on overshooting, which are falsifiable outside any internal definition. The computational-reduction scheme is presented as an engineering choice rather than a load-bearing mathematical step. This is a standard non-circular empirical ML paper.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The central claim rests on the unstated premise that the generative model can be trained to produce representations whose stability directly causes the observed RL gains; no free parameters, axioms, or invented entities are explicitly listed in the abstract.

pith-pipeline@v0.9.0 · 5702 in / 1061 out tokens · 29175 ms · 2026-05-25T18:54:11.645233+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Dream to Control: Learning Behaviors by Latent Imagination

    cs.LG 2019-12 accept novelty 7.0

    Dreamer learns to control from images by imagining and optimizing behaviors in a learned latent world model, outperforming prior methods on 20 visual tasks in data efficiency and final performance.

Reference graph

Works this paper leans on

65 extracted references · 65 canonical work pages · cited by 1 Pith paper · 27 internal anchors

  1. [1]

    Representation learning: A review and new perspectives

    Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798– 1828, 2013

    work page 2013

  2. [2]

    Predictive coding–i

    Peter Elias. Predictive coding–i. IRE Transactions on Information Theory, 1(1):16–24, 1955

    work page 1955

  3. [3]

    Curious model-building control systems

    Jürgen Schmidhuber. Curious model-building control systems. In [Proceedings] 1991 IEEE International Joint Conference on Neural Networks, pages 1458–1463. IEEE, 1991

    work page 1991

  4. [4]

    Planning and acting in partially observable stochastic domains

    Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99–134, 1998

    work page 1998

  5. [5]

    Optimal control of markov processes with incomplete state information

    Karl J Astrom. Optimal control of markov processes with incomplete state information. Journal of mathematical analysis and applications, 10(1):174–205, 1965

    work page 1965

  6. [6]

    A tutorial on hidden markov models and selected applications in speech recognition

    Lawrence R Rabiner. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257–286, 1989

    work page 1989

  7. [7]

    Reinforcement learning algorithm for partially observable markov decision problems

    Tommi Jaakkola, Satinder P Singh, and Michael I Jordan. Reinforcement learning algorithm for partially observable markov decision problems. In Advances in neural information processing systems, pages 345–352, 1995

    work page 1995

  8. [8]

    Value-function approximations for partially observable markov decision processes

    Milos Hauskrecht. Value-function approximations for partially observable markov decision processes. Journal of artificial intelligence research, 13:33–94, 2000

    work page 2000

  9. [9]

    Temporal Difference Variational Auto-Encoder

    Karol Gregor and Frederic Besse. Temporal difference variational auto-encoder. CoRR, abs/1806.03107, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  10. [10]

    Neural belief states for partially observed domains

    Pol Moreno, Jan Humplik, George Papamakarios, Bernardo Avila Pires, Lars Buesing, Nicolas Heess, and Theophane Weber. Neural belief states for partially observed domains. NeurIPS 2018 workshop on Reinforcement Learning under Partial Observability, 2018

    work page 2018

  11. [11]

    Recurrent Environment Simulators

    Silvia Chiappa, Sébastien Racanière, Daan Wierstra, and Shakir Mohamed. Recurrent environ- ment simulators. CoRR, abs/1704.02254, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  12. [12]

    Generative Temporal Models with Spatial Memory for Partially Observed Environments

    Marco Fraccaro, Danilo Jimenez Rezende, Yori Zwols, Alexander Pritzel, SM Eslami, and Fabio Viola. Generative temporal models with spatial memory for partially observed environments. arXiv preprint arXiv:1804.09401, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  13. [13]

    Video pixel networks

    Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. InProceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1771–1779. JMLR. org, 2017

    work page 2017

  14. [14]

    Action- conditional video prediction using deep networks in atari games

    Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action- conditional video prediction using deep networks in atari games. In Advances in neural information processing systems, pages 2863–2871, 2015

    work page 2015

  15. [15]

    Neural scene representation and rendering

    SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 360(6394):1204–1210, 2018

    work page 2018

  16. [16]

    Long short-term memory

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997

    work page 1997

  17. [17]

    The Kanerva Machine: A Generative Distributed Memory

    Yan Wu, Greg Wayne, Alex Graves, and Timothy Lillicrap. The kanerva machine: A generative distributed memory. arXiv preprint arXiv:1804.01756, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  18. [18]

    Optimizing Agent Behavior over Long Time Scales by Transporting Value

    Chia-Chun Hung, Timothy Lillicrap, Josh Abramson, Yan Wu, Mehdi Mirza, Federico Carnevale, Arun Ahuja, and Greg Wayne. Optimizing agent behavior over long time scales by transporting value. arXiv preprint arXiv:1810.06721, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  19. [19]

    IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

    Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, V olodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561, 2018. 10

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  20. [20]

    Pires, Toby Pohlen, and Rémi Munos

    Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Bernardo A. Pires, Toby Pohlen, and Rémi Munos. Neural predictive belief representations. CoRR, abs/1811.06407, 2018

    work page arXiv 2018

  21. [21]

    Towards conceptual compression

    Karol Gregor, Frederic Besse, Danilo Jimenez Rezende, Ivo Danihelka, and Daan Wierstra. Towards conceptual compression. In Advances In Neural Information Processing Systems , pages 3549–3557, 2016

    work page 2016

  22. [22]

    Conditional Adversarial Generative Flow for Controllable Image Synthesis

    Rui Liu, Yu Liu, Xinyu Gong, Xiaogang Wang, and Hongsheng Li. Conditional adversarial generative flow for controllable image synthesis. arXiv preprint arXiv:1904.01782, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1904

  23. [23]

    Fixing a Broken ELBO

    Alexander A Alemi, Ben Poole, Ian Fischer, Joshua V Dillon, Rif A Saurous, and Kevin Murphy. Fixing a broken elbo. arXiv preprint arXiv:1711.00464, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  24. [24]

    PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications

    Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications.arXiv preprint arXiv:1701.05517, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  25. [25]

    Preventing Posterior Collapse with delta-VAEs

    Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta-vaes. arXiv preprint arXiv:1901.03416, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1901

  26. [26]

    Conditional image generation with pixelcnn decoders

    Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelcnn decoders. In Advances in neural information processing systems, pages 4790–4798, 2016

    work page 2016

  27. [27]

    Density estimation using Real NVP

    Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  28. [28]

    Generalized elbo with constrained optimization, geco

    Danilo Jimenez Rezende and Fabio Viola. Generalized elbo with constrained optimization, geco. 2018

    work page 2018

  29. [29]

    Learning and Querying Fast Generative Models for Reinforcement Learning

    Lars Buesing, Theophane Weber, Sebastien Racaniere, SM Eslami, Danilo Rezende, David P Reichert, Fabio Viola, Frederic Besse, Karol Gregor, Demis Hassabis, et al. Learning and querying fast generative models for reinforcement learning. arXiv preprint arXiv:1802.03006, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  30. [30]

    World Models

    David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  31. [31]

    Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings

    John D Co-Reyes, YuXuan Liu, Abhishek Gupta, Benjamin Eysenbach, Pieter Abbeel, and Sergey Levine. Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory embeddings. arXiv preprint arXiv:1806.02813, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  32. [32]

    Learning Latent Dynamics for Planning from Pixels

    Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  33. [33]

    van Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Arnold, David P

    David Silver, Hado P. van Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Arnold, David P. Reichert, Neil C. Rabinowitz, André Barreto, and Thomas Degris. The predictron: End-to-end learning and planning. In ICML, 2017

    work page 2017

  34. [34]

    Learning Awareness Models

    Brandon Amos, Laurent Dinh, Serkan Cabi, Thomas Rothörl, Sergio Gómez Colmenarejo, Alistair Muldal, Tom Erez, Yuval Tassa, Nando de Freitas, and Misha Denil. Learning awareness models. arXiv preprint arXiv:1804.06318, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  35. [35]

    Unsupervised learning of depth and ego-motion from video

    Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1851–1858, 2017

    work page 2017

  36. [36]

    Geometric consistency for self-supervised end-to-end visual odometry

    Ganesh Iyer, J Krishna Murthy, Gunshi Gupta, Madhava Krishna, and Liam Paull. Geometric consistency for self-supervised end-to-end visual odometry. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 267–275, 2018

    work page 2018

  37. [37]

    Neural SLAM: Learning to explore with external memory.arXiv preprint arXiv:1706.09520,

    Jingwei Zhang, Lei Tai, Joschka Boedecker, Wolfram Burgard, and Ming Liu. Neural slam: Learning to explore with external memory. arXiv preprint arXiv:1706.09520, 2017. 11

    work page arXiv 2017

  38. [38]

    Neural map: Structured memory for deep reinforce- ment learning

    Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforce- ment learning. In International Conference on Learning Representations, 2018

    work page 2018

  39. [39]

    Navigation and planning in latent maps

    Baris Kayalibay, Atanas Mirchev, Maximilian Soelch, Patrick van der Smagt, and Justin Bayer. Navigation and planning in latent maps. 2018

    work page 2018

  40. [40]

    Cogni- tive mapping and planning for visual navigation

    Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cogni- tive mapping and planning for visual navigation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2616–2625, 2017

    work page 2017

  41. [41]

    Formal theory of creativity, fun, and intrinsic motivation (1990–2010)

    Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990–2010). IEEE Transactions on Autonomous Mental Development, 2(3):230–247, 2010

    work page 1990

  42. [42]

    Model-based reinforcement learning with parametrized physical models and optimism-driven exploration

    Chris Xie, Sachin Patil, Teodor Moldovan, Sergey Levine, and Pieter Abbeel. Model-based reinforcement learning with parametrized physical models and optimism-driven exploration. In 2016 IEEE international conference on robotics and automation (ICRA), pages 504–511. IEEE, 2016

    work page 2016

  43. [43]

    A dual back-propagation scheme for scalar reward learning

    Paul Munro. A dual back-propagation scheme for scalar reward learning. In Ninth Annual Conference of the Cognitive Science Society, pages 165–176, 1987

    work page 1987

  44. [44]

    Learning how the world works: Specifications for predictive networks in robots and brains

    Paul J Werbos. Learning how the world works: Specifications for predictive networks in robots and brains. In Proceedings of IEEE International Conference on Systems, Man and Cybernetics, NY, 1987

    work page 1987

  45. [45]

    The truck backer-upper: An example of self-learning in neural networks

    Derrick Nguyen and Bernard Widrow. The truck backer-upper: An example of self-learning in neural networks. In Advanced neural computers, pages 11–19. Elsevier, 1990

    work page 1990

  46. [46]

    Rasmussen

    Marc Peter Deisenroth and Carl E. Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In ICML, 2011

    work page 2011

  47. [47]

    Memory approaches to reinforcement learning in non- Markovian domains

    Long-Ji Lin and Tom M Mitchell. Memory approaches to reinforcement learning in non- Markovian domains. Citeseer, 1992

    work page 1992

  48. [48]

    Recurrent Reinforcement Learning: A Hybrid Approach

    Xiujun Li, Lihong Li, Jianfeng Gao, Xiaodong He, Jianshu Chen, Li Deng, and Ji He. Recurrent reinforcement learning: a hybrid approach. arXiv preprint arXiv:1509.03044, 2015

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  49. [49]

    An object-oriented representation for efficient reinforcement learning

    Carlos Diuk, Andre Cohen, and Michael L Littman. An object-oriented representation for efficient reinforcement learning. InProceedings of the 25th international conference on Machine learning, pages 240–247. ACM, 2008

    work page 2008

  50. [50]

    Deep Variational Reinforcement Learning for POMDPs

    Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational reinforcement learning for POMDPs. arXiv preprint arXiv:1806.02426, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  51. [51]

    Learning to Navigate in Complex Environments

    Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navigate in complex environments. arXiv preprint arXiv:1611.03673, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  52. [52]

    Playing fps games with deep reinforcement learning

    Guillaume Lample and Devendra Singh Chaplot. Playing fps games with deep reinforcement learning. In Thirty-First AAAI Conference on Artificial Intelligence, 2017

    work page 2017

  53. [53]

    Reinforcement Learning with Unsupervised Auxiliary Tasks

    Max Jaderberg, V olodymyr Mnih, Wojciech Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. CoRR, abs/1611.05397, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  54. [54]

    Representation Learning with Contrastive Predictive Coding

    Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1807.03748, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  55. [55]

    Darla: Improving zero-shot transfer in reinforcement learning

    Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot transfer in reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1480–1490. JMLR. org, 2017. 12

    work page 2017

  56. [56]

    Imagination-augmented agents for deep reinforcement learning

    Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomenech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al. Imagination-augmented agents for deep reinforcement learning. In Advances in neural information processing systems, pages 5690–5701, 2017

    work page 2017

  57. [57]

    Learning to Act by Predicting the Future

    Alexey Dosovitskiy and Vladlen Koltun. Learning to act by predicting the future.arXiv preprint arXiv:1611.01779, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  58. [58]

    Successor features for transfer in reinforcement learning

    André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In Advances in neural information processing systems, pages 4055–4065, 2017

    work page 2017

  59. [59]

    Improving generalization for temporal difference learning: The successor repre- sentation

    Peter Dayan. Improving generalization for temporal difference learning: The successor repre- sentation. Neural Computation, 5(4):613–624, 1993

    work page 1993

  60. [60]

    Deep Successor Reinforcement Learning

    Tejas D Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J Gershman. Deep successor reinforcement learning. arXiv preprint arXiv:1606.02396, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  61. [61]

    Adam: A Method for Stochastic Optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  62. [62]

    DeepMind Lab

    Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab.arXiv preprint arXiv:1612.03801, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  63. [63]

    Auto-Encoding Variational Bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

    work page internal anchor Pith review Pith/arXiv arXiv 2013

  64. [64]

    Stochastic Backpropagation and Approximate Inference in Deep Generative Models

    Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082, 2014

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  65. [65]

    Learning attractor dynamics for generative memory

    Yan Wu, Gregory Wayne, Karol Gregor, and Timothy Lillicrap. Learning attractor dynamics for generative memory. In Advances in Neural Information Processing Systems, pages 9379–9388, 2018. 13 Appendix A Modified ConvDRA W Convolutional DRAW [21] is an instance of a deep variational auto-encoder [63, 64]. It has a recurrent encoder and recurrent decoder with...

    work page 2018