pith. sign in

arxiv: 1906.08649 · v1 · pith:NYJHYOBHnew · submitted 2019-06-20 · 💻 cs.LG · cs.AI· cs.RO· stat.ML

Exploring Model-based Planning with Policy Networks

Tingwu Wang , Jimmy Ba This is my paper

Pith reviewed 2026-05-25 19:44 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.ROstat.ML
keywords model-based reinforcement learningpolicy networksonline planningsample efficiencyMuJoCocontinuous controlmodel predictive control
0
0 comments X

The pith

Optimizing planning over policy networks inside a dynamics model yields state-of-the-art sample efficiency on MuJoCo tasks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes POPLIN, a model-based reinforcement learning method that formulates planning as optimization over a policy network rather than random sampling in action space. At each step the algorithm either optimizes action sequences initialized from the policy or optimizes the policy parameters directly, all inside a learned dynamics model. This produces policies that require roughly three times fewer environment samples than prior methods such as PETS, TD3 and SAC while reaching higher final performance. The authors further observe that the optimization surface is smoother when working in parameter space than in raw action space. In some environments the resulting policy network can be used at test time without continued model-predictive control.

Core claim

Formulating each planning step as an optimization problem over a policy network—either by refining action sequences that the network proposes or by directly adjusting the network parameters—inside the learned dynamics model produces action sequences that transfer to the real environment more effectively than random search in action space.

What carries the argument

Policy network used to initialize or directly parameterize the optimization of actions inside the learned dynamics model at every time step.

If this is right

  • Planning becomes more efficient in high-dimensional continuous action spaces because the policy network supplies a structured starting point or parameterization.
  • The smoother optimization landscape in parameter space reduces the number of samples needed to reach high-performing policies.
  • For some locomotion tasks the distilled policy can be deployed directly without repeated online planning at test time.
  • The same planning procedure can be applied on top of any differentiable dynamics model that supports gradient-based optimization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Policy networks may act as a useful regularizer that keeps planned trajectories within regions where the model is more reliable.
  • The approach could be combined with ensemble or uncertainty-aware dynamics models to further extend the reliable planning horizon.
  • Similar parameter-space planning might improve efficiency in other sequential decision problems where an approximate model exists but exhaustive search is intractable.

Load-bearing premise

The learned dynamics model must stay accurate enough over the chosen planning horizon for the optimized actions or parameters to produce useful behavior when executed in the real environment.

What would settle it

An experiment that measures model prediction error over the planning horizon and shows that the reported performance gains disappear once that error exceeds a modest threshold while all other algorithmic choices remain fixed.

Figures

Figures reproduced from arXiv: 1906.08649 by Jimmy Ba, Tingwu Wang.

Figure 1
Figure 1. Figure 1: We transform each planned candidate action trajectory with PCA into a 2D blue scatter. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Performance curves of POPLIN-P, POPLIN-A and other state-of-the-art algorithms on [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The MPC control and policy control performance of the proposed POPLIN-A, and POPLIN [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The performance of PETS, POPLIN-A, POPLIN-P using different population size of [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The reward optimization surface in the solution space. The expected reward is higher from [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: The performance of POPLIN-A, POPLIN-P-BC, POPLIN-P-Avg, POPLIN-P-GAN using [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 6
Figure 6. Figure 6: The action distribu￾tion in a episode visualized in the projected 2D PCA space. 5.4 Ablation Study In this section, we study how sensitive our algorithms are with respect to some of the crucial hyper-parameters, for example, the initial variance of the CEM noise distribution. We also show the performance of different algorithm variants. The full ablation study and performance against different random seeds… view at source ↗
Figure 8
Figure 8. Figure 8: Full Performance of POPLIN-P, POPLIN-A and other state-of-the-art algorithms on 12 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The planning performance and the testing performance of the proposed POPLIN-A, and [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: The performance of POPLIN-A, POPLIN-P-BC, POPLIN-P-Avg, POPLIN-P-GAN using [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: The performance of POPLIN-A, POPLIN-P, and PETS of different random seeds. [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: The performance of PETS, POPLIN-A, POPLIN-P-Avg, POPLIN-P-BC and POPLIN [PITH_FULL_IMAGE:figures/full_fig_p017_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Reward surface in solution space (action space) for PETS algorithm. [PITH_FULL_IMAGE:figures/full_fig_p018_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Reward surface in solution space (action space) for POPLIN-A-Replan. [PITH_FULL_IMAGE:figures/full_fig_p018_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Reward surface in solution space (action space) for POPLIN-A-Init. [PITH_FULL_IMAGE:figures/full_fig_p018_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Reward surface in solution space (parameter space) for POPLIN-P with 0 hidden layer. [PITH_FULL_IMAGE:figures/full_fig_p019_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Reward surface in solution space (parameter space) for POPLIN-P using 1 hidden layer. [PITH_FULL_IMAGE:figures/full_fig_p019_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: The color indicates the expected cost (negative of expected reward). We emphasis that all [PITH_FULL_IMAGE:figures/full_fig_p019_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: The figures are the planned trajectories of PETS. [PITH_FULL_IMAGE:figures/full_fig_p020_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: The figures are the planned trajectories of POPLIN-P using 1 hidden layer MLP. [PITH_FULL_IMAGE:figures/full_fig_p020_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: The figures are the planned trajectories of POPLIN-P using 0 hidden layer MLP. [PITH_FULL_IMAGE:figures/full_fig_p020_21.png] view at source ↗
read the original abstract

Model-based reinforcement learning (MBRL) with model-predictive control or online planning has shown great potential for locomotion control tasks in terms of both sample efficiency and asymptotic performance. Despite their initial successes, the existing planning methods search from candidate sequences randomly generated in the action space, which is inefficient in complex high-dimensional environments. In this paper, we propose a novel MBRL algorithm, model-based policy planning (POPLIN), that combines policy networks with online planning. More specifically, we formulate action planning at each time-step as an optimization problem using neural networks. We experiment with both optimization w.r.t. the action sequences initialized from the policy network, and also online optimization directly w.r.t. the parameters of the policy network. We show that POPLIN obtains state-of-the-art performance in the MuJoCo benchmarking environments, being about 3x more sample efficient than the state-of-the-art algorithms, such as PETS, TD3 and SAC. To explain the effectiveness of our algorithm, we show that the optimization surface in parameter space is smoother than in action space. Further more, we found the distilled policy network can be effectively applied without the expansive model predictive control during test time for some environments such as Cheetah. Code is released in https://github.com/WilsonWangTHU/POPLIN.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript proposes POPLIN, a model-based RL method that formulates online planning as optimization over action sequences initialized from a policy network or directly over policy parameters. It reports state-of-the-art results on MuJoCo locomotion tasks, claiming approximately 3x greater sample efficiency than PETS, TD3, and SAC; attributes gains to a smoother optimization landscape in parameter space; shows that a distilled policy can sometimes be deployed without MPC at test time; and releases code.

Significance. If the empirical claims hold after verification of model fidelity, the work would usefully demonstrate that parameter-space planning can outperform pure action-space search in MBRL while retaining the sample-efficiency advantages of model-based methods. The open-source code is a clear strength that enables direct reproduction and extension.

major comments (2)
  1. [Abstract] Abstract: the central claim that POPLIN is 'about 3x more sample efficient' than PETS, TD3, and SAC is load-bearing for the contribution yet is presented without reported multi-step dynamics-model error, held-out trajectory prediction accuracy, planning-horizon length, or statistical significance tests on the performance differences.
  2. [Abstract] Abstract / experiments: the transfer assumption that optimizing inside the learned model produces actions that succeed in the real environment is invoked without any reported planned-vs-executed discrepancy or compounding-error diagnostics on the MuJoCo tasks; this directly affects whether the 3x efficiency gain can be attributed to the planning component.
minor comments (1)
  1. [Abstract] Abstract: 'Further more' should be 'Furthermore'.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We agree that additional empirical details will strengthen the presentation of our results and will revise the manuscript accordingly. Our point-by-point responses follow.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that POPLIN is 'about 3x more sample efficient' than PETS, TD3, and SAC is load-bearing for the contribution yet is presented without reported multi-step dynamics-model error, held-out trajectory prediction accuracy, planning-horizon length, or statistical significance tests on the performance differences.

    Authors: We agree these details should be reported. The planning horizon length is 10 steps for POPLIN, PETS, and the model-free baselines (Section 4.1). We will add multi-step model prediction error and held-out trajectory accuracy metrics in the revised version. For statistical significance, the learning curves already aggregate 5 seeds with standard-deviation shading; we will add explicit discussion of the performance gaps in the text and caption. revision: yes

  2. Referee: [Abstract] Abstract / experiments: the transfer assumption that optimizing inside the learned model produces actions that succeed in the real environment is invoked without any reported planned-vs-executed discrepancy or compounding-error diagnostics on the MuJoCo tasks; this directly affects whether the 3x efficiency gain can be attributed to the planning component.

    Authors: All reported returns are obtained by executing the first planned action in the true MuJoCo environment at every step (standard MPC procedure). The sample-efficiency comparison therefore already reflects real-environment performance. To address the request for explicit diagnostics, we will add planned-versus-executed trajectory discrepancy plots over the horizon in the appendix of the revision. revision: yes

Circularity Check

0 steps flagged

No circularity; empirical algorithm with benchmark results

full rationale

The paper introduces the POPLIN algorithm combining policy networks with model-based planning and reports empirical results on MuJoCo environments showing improved sample efficiency over baselines. No derivation chain, first-principles prediction, or uniqueness theorem is claimed. Performance claims rest on external benchmark comparisons rather than any quantity fitted inside the paper and then renamed as a prediction. No self-definitional steps, fitted-input predictions, or load-bearing self-citations appear in the provided text. The central result is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The method inherits standard RL assumptions (Markov decision process, differentiable dynamics model) but introduces no new free parameters or invented entities visible in the abstract; the central empirical claim rests on the unstated premise that the learned model is accurate enough for the reported planning horizon.

axioms (1)
  • domain assumption The environment can be modeled by a differentiable dynamics function that remains valid over the multi-step planning horizon.
    Implicit in any model-based planning claim; required for the optimization to produce transferable actions.

pith-pipeline@v0.9.0 · 5762 in / 1251 out tokens · 18263 ms · 2026-05-25T19:44:56.064491+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. WOMBET: World Model-Based Experience Transfer for Robust and Sample-efficient Reinforcement Learning

    cs.LG 2026-04 unverdicted novelty 7.0

    WOMBET generates reliable prior data with world-model uncertainty penalization and transfers it to target tasks via adaptive offline-online sampling, yielding better sample efficiency than baselines.

  2. Mastering Atari with Discrete World Models

    cs.LG 2020-10 accept novelty 7.0

    DreamerV2 reaches human-level performance on 55 Atari games by learning behaviors inside a separately trained discrete-latent world model.

  3. Dream to Control: Learning Behaviors by Latent Imagination

    cs.LG 2019-12 accept novelty 7.0

    Dreamer learns to control from images by imagining and optimizing behaviors in a learned latent world model, outperforming prior methods on 20 visual tasks in data efficiency and final performance.

  4. MODIP: Efficient Model-Based Optimization for Diffusion Policies

    cs.LG 2026-06 unverdicted novelty 6.0

    MODIP fine-tunes diffusion policies offline-to-online by training a world model, running MPC with terminal state values inside it to create targets, and using policy-independent TD critics, yielding gains over BC on D...

  5. Hyperfastrl: Hypernetwork-based reinforcement learning for unified control of parametric chaotic PDEs

    cs.CE 2026-04 unverdicted novelty 6.0

    Hypernetworks map a forcing parameter directly to policy weights in an RL framework, enabling unified stabilization of the Kuramoto-Sivashinsky equation across regimes with KAN architectures showing strongest extrapolation.

Reference graph

Works this paper leans on

43 extracted references · 43 canonical work pages · cited by 5 Pith papers · 16 internal anchors

  1. [1]

    The cross-entropy method for optimization

    Zdravko I Botev, Dirk P Kroese, Reuven Y Rubinstein, and Pierre L’Ecuyer. The cross-entropy method for optimization. In Handbook of statistics, volume 31, pages 35–59. Elsevier, 2013

    work page 2013

  2. [2]

    OpenAI Gym

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  3. [3]

    Sample- efficient reinforcement learning with stochastic ensemble value expansion

    Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In Advances in Neural Information Processing Systems, pages 8224–8234, 2018

    work page 2018

  4. [4]

    Combining model-based and model-free updates for trajectory-centric reinforcement learning

    Yevgen Chebotar, Karol Hausman, Marvin Zhang, Gaurav Sukhatme, Stefan Schaal, and Sergey Levine. Combining model-based and model-free updates for trajectory-centric reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-V olume 70, pages 703–711. JMLR. org, 2017

    work page 2017

  5. [5]

    Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

    Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforce- ment learning in a handful of trials using probabilistic dynamics models. arXiv preprint arXiv:1805.12114, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  6. [6]

    A tutorial on the cross-entropy method

    Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial on the cross-entropy method. Annals of operations research, 134(1):19–67, 2005

    work page 2005

  7. [7]

    Pilco: A model-based and data-efficient approach to policy search

    Marc Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11), pages 465–472, 2011

    work page 2011

  8. [8]

    Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning

    Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv:1803.00101, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  9. [9]

    Addressing Function Approximation Error in Actor-Critic Methods

    Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1802.09477, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  10. [10]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014

    work page 2014

  11. [11]

    Continuous deep q-learning with model-based acceleration

    Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International Conference on Machine Learning , pages 2829–2838, 2016

    work page 2016

  12. [12]

    Recurrent world models facilitate policy evolution

    David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems , pages 2450–2462, 2018

    work page 2018

  13. [13]

    World Models

    David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  14. [14]

    Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  15. [15]

    Learning Latent Dynamics for Planning from Pixels

    Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  16. [16]

    Emergence of Locomotion Behaviours in Rich Environments

    Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Ali Eslami, Martin Riedmiller, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  17. [17]

    Learning continuous control policies by stochastic value gradients

    Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, pages 2944–2952, 2015

    work page 2015

  18. [18]

    arXiv preprint arXiv:1903.00374 , year=

    Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model- based reinforcement learning for atari. arXiv preprint arXiv:1903.00374, 2019

    work page arXiv 1903

  19. [19]

    Auto-Encoding Variational Bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 9

    work page internal anchor Pith review Pith/arXiv arXiv 2013

  20. [20]

    Model-Ensemble Trust-Region Policy Optimization

    Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv:1802.10592, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  21. [21]

    Learning neural network policies with guided policy search under unknown dynamics

    Sergey Levine and Pieter Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems , pages 1071–1079, 2014

    work page 2014

  22. [22]

    End-to-end training of deep visuomotor policies

    Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research , 17(1):1334–1373, 2016

    work page 2016

  23. [23]

    Guided policy search

    Sergey Levine and Vladlen Koltun. Guided policy search. In International Conference on Machine Learning, pages 1–9, 2013

    work page 2013

  24. [24]

    Visualizing the loss landscape of neural nets

    Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems , pages 6389–6399, 2018

    work page 2018

  25. [25]

    Iterative linear quadratic regulator design for nonlinear biological movement systems

    Weiwei Li and Emanuel Todorov. Iterative linear quadratic regulator design for nonlinear biological movement systems. In ICINCO (1), pages 222–229, 2004

    work page 2004

  26. [26]

    Continuous control with deep reinforcement learning

    Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  27. [27]

    Algo- rithmic framework for model-based deep reinforcement learning with theoretical guarantees

    Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algo- rithmic framework for model-based deep reinforcement learning with theoretical guarantees. ICLR, 2019

    work page 2019

  28. [28]

    Playing Atari with Deep Reinforcement Learning

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013

    work page internal anchor Pith review Pith/arXiv arXiv 2013

  29. [29]

    Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning

    Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. arXiv preprint arXiv:1708.02596, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  30. [30]

    The loss surface of deep and wide neural networks

    Quynh Nguyen and Matthias Hein. The loss surface of deep and wide neural networks. In Proceedings of the 34th International Conference on Machine Learning-V olume 70 , pages 2603–2612. JMLR. org, 2017

    work page 2017

  31. [31]

    Robust constrained model predictive control

    Arthur George Richards. Robust constrained model predictive control . PhD thesis, Mas- sachusetts Institute of Technology, 2005

    work page 2005

  32. [32]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems , pages 2234–2242, 2016

    work page 2016

  33. [33]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 1889–1897, 2015

    work page 2015

  34. [34]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  35. [35]

    Mas- tering the game of go with deep neural networks and tree search

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016

    work page 2016

  36. [36]

    Mastering the game of go without human knowledge

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354–359, 2017

    work page 2017

  37. [37]

    Exponentially vanishing sub-optimal local minima in multilayer neural networks

    Daniel Soudry and Elad Hoffer. Exponentially vanishing sub-optimal local minima in multilayer neural networks. arXiv preprint arXiv:1702.05777, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  38. [38]

    Integrated architectures for learning, planning, and reacting based on approximating dynamic programming

    Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine Learning Proceedings 1990, pages 216–224. Elsevier, 1990

    work page 1990

  39. [39]

    Dyna, an integrated architecture for learning, planning, and reacting

    Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin, 2(4):160–163, 1991. 10

    work page 1991

  40. [40]

    Synthesis and stabilization of complex behaviors through online trajectory optimization

    Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. InIntelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pages 4906–4913. IEEE, 2012

    work page 2012

  41. [41]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on , pages 5026–5033. IEEE, 2012

    work page 2012

  42. [42]

    A generalized iterative lqg method for locally-optimal feedback control of constrained nonlinear stochastic systems

    Emanuel Todorov and Weiwei Li. A generalized iterative lqg method for locally-optimal feedback control of constrained nonlinear stochastic systems. In Proceedings of the 2005, American Control Conference, 2005., pages 300–306. IEEE, 2005

    work page 2005

  43. [43]

    SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning

    Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew J Johnson, and Sergey Levine. Solar: Deep structured latent representations for model-based reinforcement learning. arXiv preprint arXiv:1808.09105, 2018. 11 A Appendix A.1 Algorithm Diagrams To better illustrate the algorithm variants of our proposed methods, we summarize them in Algo- rith...

    work page internal anchor Pith review Pith/arXiv arXiv 2018