pith. sign in

arxiv: 2605.02405 · v3 · pith:MM7HFCVLnew · submitted 2026-05-04 · 💻 cs.LG

Closed-Loop CO2 Storage Control With History-Based Reinforcement Learning and Latent Model-Based Adaptation

Sofianos Panagiotis Fotias , Vassilis Gaganis This is my paper

Pith reviewed 2026-06-30 23:54 UTC · model grok-4.3

classification 💻 cs.LG
keywords closed-loop CO2 storagereinforcement learninghistory-conditioned policieslatent model adaptationpartially observable controlreservoir simulationinjector failureleakage
0
0 comments X

The pith

History-conditioned reinforcement learning policies recover nearly all privileged-state performance in CO2 storage control using only well-level observations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper formulates CO2 injection and brine production control as a partially observable sequential decision problem and trains deep reinforcement-learning controllers on high-fidelity reservoir simulations. It compares privileged-state, well-only, history-conditioned, masking-curriculum, and asymmetric teacher-student policies to measure the value of temporal well-response information. It then tests a latent model-based adaptation pipeline that reuses nominal latent dynamics and retunes controllers for injector failure, leakage-induced dynamics, reward shifts, and compartmentalized connectivity. The work shows that history-conditioned policies achieve nearly the same performance as privileged-state ones while relying only on deployable well data, and that latent model-based retuning outperforms direct model-free retuning under fixed scenario-specific simulator budgets in abnormal cases.

Core claim

History-conditioned policies recover nearly all of the privileged-state performance while using only deployable well-level information, and latent model-based retuning outperforms direct model-free retuning under the same scenario-specific real-simulator budget in the abnormal operating cases. The proposed framework therefore provides a simulator-budget-aware alternative to repeated online history matching and re-optimization for closed-loop CO2 storage control.

What carries the argument

History-conditioned deep reinforcement learning policies combined with a latent model-based adaptation pipeline that reuses nominal latent dynamics for retuning under abnormal scenarios.

If this is right

  • Controllers can be trained and deployed using only measurements realistically available at wells rather than full reservoir states.
  • Abnormal operating conditions such as injector failure or leakage can be handled by retuning on a limited additional simulator budget instead of full retraining.
  • The approach reduces dependence on repeated online history matching for adapting control policies to changing reservoir behavior.
  • Training-time access to simulator states improves policy quality but is not required at deployment time.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The latent adaptation step may generalize to other subsurface flow control tasks where simulator budgets are limited and dynamics shift occur.
  • If the performance gap between history-conditioned and privileged policies remains small across more reservoir models, it would support wider use of partial-observation RL in geological storage.
  • The budget comparison between latent and model-free retuning could motivate similar hybrid adaptation pipelines in other engineering domains with expensive simulators.

Load-bearing premise

High-fidelity reservoir simulations accurately represent the uncertain real-world reservoir dynamics, including the specific abnormal scenarios tested.

What would settle it

Deploy the trained history-conditioned and latent-adapted controllers in a real CO2 storage site experiencing injector failure or leakage and measure whether achieved injection rates, pressure management, and leakage containment match the simulated performance within a specified tolerance.

Figures

Figures reproduced from arXiv: 2605.02405 by Sofianos Panagiotis Fotias, Vassilis Gaganis.

Figure 1
Figure 1. Figure 1: Closed-loop CCS control formulation considered in this work. The reservoir simulator evolves view at source ↗
Figure 2
Figure 2. Figure 2: Training-deployment setting considered in this work. Policies are trained on ensembles of prior view at source ↗
Figure 3
Figure 3. Figure 3: Baseline information regimes. The privileged-state benchmark uses dense simulator fields and view at source ↗
Figure 4
Figure 4. Figure 4: History-conditioned model. Current well observations and a rolling history of public well view at source ↗
Figure 5
Figure 5. Figure 5: Masked-critic curriculum. During training, the critic receives progressively masked spatial sim view at source ↗
Figure 6
Figure 6. Figure 6: Asymmetric teacher-student model. During training, privileged teacher critics use dense spatial view at source ↗
Figure 7
Figure 7. Figure 7: Model-based pipeline used in this work. Public observations are mapped to a deployable latent view at source ↗
Figure 8
Figure 8. Figure 8: Scenario 1 methodology. The actor outputs an 11-dimensional nominal action, after which the view at source ↗
Figure 9
Figure 9. Figure 9: Residual world-model adaptation for Scenarios 2 and 3. Abnormal latent transitions are encoded view at source ↗
Figure 10
Figure 10. Figure 10: Test return as a function of training epoch for the five model-free variants. The well-only base view at source ↗
Figure 11
Figure 11. Figure 11: Scenario 0: nominal model-based retention. Real-environment evaluation return as a function view at source ↗
Figure 12
Figure 12. Figure 12: Scenario 1: adaptation under known injector failure. Real-environment evaluation return as a view at source ↗
Figure 13
Figure 13. Figure 13: Scenario 2: adaptation under leakage-induced dynamics and reward shift. Real-environment view at source ↗
Figure 14
Figure 14. Figure 14: Scenario 3: adaptation under compartmentalized connectivity shift. Real-environment eval view at source ↗
read the original abstract

Closed-loop management of geological CO2 storage requires control policies that adapt to uncertain reservoir behavior while relying on observations that are realistically available during operation. This work formulates CO2 injection and brine-production control as a partially observable sequential decision problem and studies deployable deep reinforcement-learning controllers trained with high-fidelity reservoir simulation. We first compare privileged-state, well-only, history-conditioned, masking-curriculum, and asymmetric teacher-student model-free policies in order to quantify the value of temporal well-response information and training-time privileged simulator states. We then evaluate a latent model-based adaptation pipeline that reuses nominal latent dynamics and retunes controllers under known injector failure, leakage-induced dynamics and reward shift, and compartmentalized reservoir connectivity. The results show that history-conditioned policies recover nearly all of the privileged-state performance while using only deployable well-level information, and that latent model-based retuning outperforms direct model-free retuning under the same scenario-specific real-simulator budget in the abnormal operating cases. The proposed framework therefore provides a simulator-budget-aware alternative to repeated online history matching and re-optimization for closed-loop CO2 storage control.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper formulates CO2 injection and brine-production control as a POMDP and compares privileged-state, well-only, history-conditioned, masking-curriculum, and asymmetric teacher-student model-free RL policies trained in high-fidelity reservoir simulators. It then evaluates a latent model-based adaptation pipeline that reuses nominal latent dynamics and retunes controllers for injector failure, leakage, and compartmentalization scenarios, claiming that history-conditioned policies recover nearly all privileged performance using only deployable observations and that latent retuning outperforms model-free retuning under fixed real-simulator budgets.

Significance. If the empirical comparisons hold under the reported conditions, the work demonstrates a practical, simulator-budget-aware alternative to repeated online history matching for closed-loop geological CO2 storage, highlighting the value of temporal history conditioning and latent adaptation in partially observable reservoir control.

major comments (3)
  1. [Abstract] Abstract and results section: comparative performance claims (history-conditioned recovering nearly all privileged performance; latent retuning outperforming model-free under fixed budget) are reported without any description of the number of independent training runs, random seeds, statistical tests, or variance estimates, which is load-bearing for assessing whether the observed differences are reliable.
  2. [Adaptation pipeline evaluation] The adaptation experiments evaluate all pipelines inside the same high-fidelity simulator family used for nominal training; no section quantifies the simulation-reality gap, uncertainty propagation from geological parameters, or out-of-distribution behavior on the abnormal scenarios (injector failure, leakage, compartmentalization), which directly limits the applicability of the headline claims to field deployment.
  3. [Methods] No details are provided on hyperparameter selection, reward-function coefficients, or training procedure for the RL agents, making it impossible to reproduce or assess sensitivity of the reported policy comparisons.
minor comments (2)
  1. [Latent model-based adaptation] Notation for the latent dynamics model and the retuning objective should be introduced with explicit equations rather than prose descriptions.
  2. [Results figures] Figure captions for policy performance plots should include the exact number of episodes or runs per bar and any error bars used.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive report. We address each major comment below and will revise the manuscript to improve clarity, reproducibility, and discussion of limitations.

read point-by-point responses

  1. Referee: [Abstract] Abstract and results section: comparative performance claims (history-conditioned recovering nearly all privileged performance; latent retuning outperforming model-free under fixed budget) are reported without any description of the number of independent training runs, random seeds, statistical tests, or variance estimates, which is load-bearing for assessing whether the observed differences are reliable.

    Authors: We agree that the absence of these details weakens the reliability assessment of the reported differences. In the revised manuscript we will add explicit reporting of the number of independent training runs (with random seeds), variance estimates across runs, and any statistical comparisons performed on the policy performance metrics. revision: yes

  2. Referee: [Adaptation pipeline evaluation] The adaptation experiments evaluate all pipelines inside the same high-fidelity simulator family used for nominal training; no section quantifies the simulation-reality gap, uncertainty propagation from geological parameters, or out-of-distribution behavior on the abnormal scenarios (injector failure, leakage, compartmentalization), which directly limits the applicability of the headline claims to field deployment.

    Authors: The experiments are performed within the simulator family, as the work focuses on a simulator-budget-aware control pipeline rather than direct field validation. We acknowledge that this limits immediate claims about field deployment. In revision we will add an explicit limitations subsection discussing the sim-to-real gap, geological parameter uncertainty, and the need for future out-of-distribution validation on real reservoir data. revision: partial

  3. Referee: [Methods] No details are provided on hyperparameter selection, reward-function coefficients, or training procedure for the RL agents, making it impossible to reproduce or assess sensitivity of the reported policy comparisons.

    Authors: We agree that these details are necessary for reproducibility. The revised manuscript will include a new appendix (or expanded methods section) that reports the hyperparameter search procedure, exact reward-function coefficients, network architectures, and full training protocol for all RL agents. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical comparisons within fixed simulator

full rationale

The paper reports direct experimental outcomes from training and evaluating RL policies (privileged-state, history-conditioned, etc.) and adaptation pipelines (latent model-based vs. model-free) inside the same high-fidelity reservoir simulator. No equations, uniqueness theorems, or fitted parameters are presented whose outputs are definitionally identical to their inputs; performance deltas are measured quantities, not algebraic identities. Self-citations, if present, are not load-bearing for the reported results. The derivation chain is therefore self-contained against the simulator benchmarks.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 0 invented entities

All results rest on the domain assumption that high-fidelity reservoir simulations are faithful proxies for real CO2 storage dynamics. RL training involves numerous unspecified hyperparameters and reward weights whose values are fitted to simulation outcomes. No new physical entities are postulated.

free parameters (2)
  • RL training hyperparameters
    Learning rates, network sizes, discount factors, and exploration parameters are standard in deep RL and must be chosen or tuned; they directly affect reported policy performance.
  • Reward function coefficients
    Weights balancing injection targets, brine production, pressure limits, and safety penalties are required to define the objective and are fitted within the simulator.
axioms (1)
  • domain assumption High-fidelity reservoir simulations accurately capture real reservoir behavior and the tested abnormal dynamics
    The entire training, comparison, and adaptation pipeline is performed inside these simulations; any mismatch with reality would invalidate transfer of the reported performance gains.

pith-pipeline@v0.9.1-grok · 5731 in / 1449 out tokens · 34706 ms · 2026-06-30T23:54:24.934867+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

77 extracted references · 3 canonical work pages · 3 internal anchors

  1. [1]

    Technologies and infrastructures underpinning future co2 value chains: A comprehensive review and comparative analysis.Renewable and Sustainable Energy Re- views, 85:46–68, 2018

    Sean M Jarvis and Sheila Samsatli. Technologies and infrastructures underpinning future co2 value chains: A comprehensive review and comparative analysis.Renewable and Sustainable Energy Re- views, 85:46–68, 2018

    2018

  2. [2]

    Paolo Gabrielli, Matteo Gazzani, and Marco Mazzotti. The role of carbon capture and utilization, carbon capture and storage, and biomass to enable a net-zero-co2 emissions chemical industry.In- dustrial & Engineering Chemistry Research, 59(15):7033–7045, 2020. 25

    2020

  3. [3]

    The role of carbon capture and storage (ccs) technologies in a net-zero carbon future

    Mai Bui, Graeme Douglas Puxty, Matteo Gazzani, Salman Masoudi Soltani, and Carlos Pozo. The role of carbon capture and storage (ccs) technologies in a net-zero carbon future. 2021

    2021

  4. [4]

    Introduction to geological storage.Carbon Capture and Storage; Elsevier: Amsterdam, The Netherlands, pages 285–304, 2017

    SA Rackley and SA Rackley. Introduction to geological storage.Carbon Capture and Storage; Elsevier: Amsterdam, The Netherlands, pages 285–304, 2017

    2017

  5. [5]

    Criteria for co2 storage in geological formations.Podzemni radovi, (32):61–74, 2018

    Lola Tomi´ c, Vesna Karovi´ c Mariˇ ci´ c, Duˇ san Danilovi´ c, and Miroslav Crnogorac. Criteria for co2 storage in geological formations.Podzemni radovi, (32):61–74, 2018

    2018

  6. [6]

    Co2 storage in deep saline aquifers

    Xiaoyan Ji and Chen Zhu. Co2 storage in deep saline aquifers. InNovel materials for carbon dioxide mitigation technology, pages 299–332. Elsevier, 2015

    2015

  7. [7]

    Review of co2 storage efficiency in deep saline aquifers.International Journal of Greenhouse Gas Control, 40:188–202, 2015

    Stefan Bachu. Review of co2 storage efficiency in deep saline aquifers.International Journal of Greenhouse Gas Control, 40:188–202, 2015

    2015

  8. [8]

    Geological storage of co2 in saline aquifers—a review of the experience from existing storage operations.International journal of greenhouse gas control, 4(4):659–667, 2010

    Karsten Michael, Alexandra Golab, Valeriya Shulakova, Jonathan Ennis-King, Guy Allinson, Sandeep Sharma, and Toby Aiken. Geological storage of co2 in saline aquifers—a review of the experience from existing storage operations.International journal of greenhouse gas control, 4(4):659–667, 2010

    2010

  9. [9]

    Co2 storage in depleted or depleting oil and gas fields: what can we learn from existing projects?Energy Procedia, 114:5680–5690, 2017

    Sarah Hannis, Jiemin Lu, Andy Chadwick, Sue Hovorka, Karen Kirk, Katherine Romanak, and Jonathan Pearce. Co2 storage in depleted or depleting oil and gas fields: what can we learn from existing projects?Energy Procedia, 114:5680–5690, 2017

    2017

  10. [10]

    Co 2-eor/sequestration: Current trends and future horizons

    Erfan Mohammadian, Badrul Mohamed Jan, Amin Azdarpour, Hossein Hamidi, Nur Hidayati Binti Othman, Aqilah Dollah, Siti Nurliyana Binti Che Mohamed Hussein, and Rozana Azrina Binti Sazali. Co 2-eor/sequestration: Current trends and future horizons. InEnhanced Oil Recovery Processes-New Technologies. IntechOpen, 2019

    2019

  11. [11]

    Co2 sequestration in depleted oil and gas reservoirs—caprock characterization and storage capacity.Energy Conversion and Management, 47(11-12):1372–1382, 2006

    Zhaowen Li, Mingzhe Dong, Shuliang Li, and Sam Huang. Co2 sequestration in depleted oil and gas reservoirs—caprock characterization and storage capacity.Energy Conversion and Management, 47(11-12):1372–1382, 2006

    2006

  12. [12]

    Carbon capture, utilization, and storage in saline aquifers: Sub- surface policies, development plans, well control strategies and optimization approaches—a review

    Ismail Ismail and Vassilis Gaganis. Carbon capture, utilization, and storage in saline aquifers: Sub- surface policies, development plans, well control strategies and optimization approaches—a review. Clean Technologies, 5(2):609–637, 2023

    2023

  13. [13]

    Code intercomparison builds confidence in numerical simulation models for geologic disposal of co2.Energy, 29(9-10):1431–1444, 2004

    Karsten Pruess, Julio Garc´ ıa, Tony Kovscek, Curt Oldenburg, Jonny Rutqvist, Carl Steefel, and Tianfu Xu. Code intercomparison builds confidence in numerical simulation models for geologic disposal of co2.Energy, 29(9-10):1431–1444, 2004

    2004

  14. [14]

    A benchmark study on problems related to co 2 storage in geologic formations: summary and discussion of the results

    Holger Class, Anozie Ebigbo, Rainer Helmig, Helge K Dahle, Jan M Nordbotten, Michael A Celia, Pascal Audigane, Melanie Darcis, Jonathan Ennis-King, Yaqing Fan, et al. A benchmark study on problems related to co 2 storage in geologic formations: summary and discussion of the results. Computational geosciences, 13:409–434, 2009

    2009

  15. [15]

    Optimal well placement and brine extraction for pressure management during co2 sequestration.International Journal of Greenhouse Gas Control, 42:175–187, 2015

    Abdullah Cihan, Jens T Birkholzer, and Marco Bianchi. Optimal well placement and brine extraction for pressure management during co2 sequestration.International Journal of Greenhouse Gas Control, 42:175–187, 2015

    2015

  16. [16]

    Optimization of well placement, co2 injection rates, and brine cycling for geological carbon sequestration.International Journal of Greenhouse Gas Control, 10:100–112, 2012

    David A Cameron and Louis J Durlofsky. Optimization of well placement, co2 injection rates, and brine cycling for geological carbon sequestration.International Journal of Greenhouse Gas Control, 10:100–112, 2012

    2012

  17. [17]

    Co2 storage in geological media: Role, means, status and barriers to deployment

    Stefan Bachu. Co2 storage in geological media: Role, means, status and barriers to deployment. Progress in energy and combustion science, 34(2):254–273, 2008. 26

    2008

  18. [18]

    The acceptability of co2 capture and storage (ccs) in europe: An assessment of the key determining factors: Part 1

    Heleen De Coninck, Todd Flach, Paul Curnow, Peter Richardson, Jason Anderson, Simon Shackley, Gudmundur Sigurthorsson, and David Reiner. The acceptability of co2 capture and storage (ccs) in europe: An assessment of the key determining factors: Part 1. scientific, technical and economic dimensions.International Journal of Greenhouse Gas Control, 3(3):333–...

    2009

  19. [19]

    Active pressure management through brine production for basin-wide deployment of geologic carbon sequestration.International Journal of Greenhouse Gas Control, 61:155–167, 2017

    Karl W Bandilla and Michael A Celia. Active pressure management through brine production for basin-wide deployment of geologic carbon sequestration.International Journal of Greenhouse Gas Control, 61:155–167, 2017

    2017

  20. [20]

    Pre-injection brine production in co2 storage reservoirs: An approach to augment the development, operation, and performance of ccs while generating water

    Thomas A Buscheck, Jeffrey M Bielicki, Joshua A White, Yunwei Sun, Yue Hao, William L Bourcier, Susan A Carroll, and Roger D Aines. Pre-injection brine production in co2 storage reservoirs: An approach to augment the development, operation, and performance of ccs while generating water. International Journal of Greenhouse Gas Control, 54:499–512, 2016

    2016

  21. [21]

    Estimating the net costs of brine production and disposal to expand pressure-limited dynamic capacity for basin-scale co2 storage in a saline formation

    Steven T Anderson and Hossein Jahediesfanjani. Estimating the net costs of brine production and disposal to expand pressure-limited dynamic capacity for basin-scale co2 storage in a saline formation. International Journal of Greenhouse Gas Control, 102:103161, 2020

    2020

  22. [22]

    Investigation of co2 storage capacity in open saline aquifers with numerical models.Procedia Engineering, 31:886–892, 2012

    Yang Wang, Yaqin Xu, and Keni Zhang. Investigation of co2 storage capacity in open saline aquifers with numerical models.Procedia Engineering, 31:886–892, 2012

    2012

  23. [23]

    Multi-objective optimization

    Kalyanmoy Deb, Karthik Sindhya, and Jussi Hakanen. Multi-objective optimization. InDecision sciences, pages 161–200. CRC Press, 2016

    2016

  24. [24]

    A holistic review on artificial intelligence techniques for well placement optimization problem.Advances in engineering software, 141:102767, 2020

    Jahedul Islam, Pandian M Vasant, Berihun Mamo Negash, Moacyr Bartholomeu Laruccia, Myo Myint, and Junzo Watada. A holistic review on artificial intelligence techniques for well placement optimization problem.Advances in engineering software, 141:102767, 2020

    2020

  25. [25]

    Learning surrogate models for simulation- based optimization.AIChE Journal, 60(6):2211–2227, 2014

    Alison Cozad, Nikolaos V Sahinidis, and David C Miller. Learning surrogate models for simulation- based optimization.AIChE Journal, 60(6):2211–2227, 2014

    2014

  26. [26]

    A Tutorial on Bayesian Optimization

    Peter I Frazier. A tutorial on bayesian optimization.arXiv preprint arXiv:1807.02811, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  27. [27]

    An introduction to continuity, extrema, and related topics for general gaussian processes

    Robert J Adler. An introduction to continuity, extrema, and related topics for general gaussian processes. IMS, 1990

    1990

  28. [28]

    Aleatory or epistemic? does it matter?Structural safety, 31(2):105–112, 2009

    Armen Der Kiureghian and Ove Ditlevsen. Aleatory or epistemic? does it matter?Structural safety, 31(2):105–112, 2009

    2009

  29. [29]

    Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods.Machine learning, 110(3):457–506, 2021

    Eyke H¨ ullermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods.Machine learning, 110(3):457–506, 2021

    2021

  30. [30]

    Optimization of well placement in carbon capture and storage (ccs): Bayesian optimization framework under permutation invariance

    Sofianos Panagiotis Fotias, Ismail Ismail, and Vassilis Gaganis. Optimization of well placement in carbon capture and storage (ccs): Bayesian optimization framework under permutation invariance. Applied Sciences, 14(8):3528, 2024

    2024

  31. [31]

    Improved reservoir management through optimal control and continuous model updating

    DR Brouwer, G Nœvdal, JD Jansen, Erland H Vefring, and CPJW Van Kruijsdijk. Improved reservoir management through optimal control and continuous model updating. InSPE Annual Technical Conference and Exhibition?, pages SPE–90149. SPE, 2004

    2004

  32. [32]

    Optimizing the performance of smart wells in com- plex reservoirs using continuously updated geological models.Journal of Petroleum Science and Engineering, 48(3-4):254–264, 2005

    Inegbenose Aitokhuehi and Louis J Durlofsky. Optimizing the performance of smart wells in com- plex reservoirs using continuously updated geological models.Journal of Petroleum Science and Engineering, 48(3-4):254–264, 2005. 27

    2005

  33. [33]

    Efficient real-time reservoir manage- ment using adjoint-based optimal control and model updating.Computational Geosciences, 10:3–36, 2006

    Pallav Sarma, Louis J Durlofsky, Khalid Aziz, and Wen H Chen. Efficient real-time reservoir manage- ment using adjoint-based optimal control and model updating.Computational Geosciences, 10:3–36, 2006

    2006

  34. [34]

    Closed-loop reservoir management

    Jan-Dirk Jansen, SD Douma, Dr R Brouwer, PMJ Van den Hof, OH Bosgra, and AW Heemink. Closed-loop reservoir management. InSPE Reservoir Simulation Conference?, pages SPE–119098. SPE, 2009

    2009

  35. [35]

    Production optimization in closed-loop reservoir management.SPE journal, 14(03):506–523, 2009

    Chunhong Wang, Gaoming Li, and Albert C Reynolds. Production optimization in closed-loop reservoir management.SPE journal, 14(03):506–523, 2009

    2009

  36. [36]

    Comprehensive framework for gradient-based optimization in closed-loop reservoir management.Computational Geosciences, 19:877–897, 2015

    Vladislav Bukshtynov, Oleg Volkov, Louis J Durlofsky, and Khalid Aziz. Comprehensive framework for gradient-based optimization in closed-loop reservoir management.Computational Geosciences, 19:877–897, 2015

    2015

  37. [37]

    A derivative-free approach for the estimation of porosity and permeability using time-lapse seismic and production data.Journal of Geophysics and Engineering, 7(4):351–368, 2010

    Mohsen Dadashpour, David Echeverria Ciaurri, Tapan Mukerji, Jon Kleppe, and Martin Landrø. A derivative-free approach for the estimation of porosity and permeability using time-lapse seismic and production data.Journal of Geophysics and Engineering, 7(4):351–368, 2010

    2010

  38. [38]

    Multilevel strategies and geological parameterizations for history matching complex reservoir models.SPE Journal, 25(01):081–104, 2020

    Yimin Liu and Louis J Durlofsky. Multilevel strategies and geological parameterizations for history matching complex reservoir models.SPE Journal, 25(01):081–104, 2020

    2020

  39. [39]

    Data assimilation for transient flow in geologic formations via ensemble kalman filter.Advances in Water Resources, 29(8):1107–1122, 2006

    Yan Chen and Dongxiao Zhang. Data assimilation for transient flow in geologic formations via ensemble kalman filter.Advances in Water Resources, 29(8):1107–1122, 2006

    2006

  40. [40]

    Ensemble smoother with multiple data assimilation

    Alexandre A Emerick and Albert C Reynolds. Ensemble smoother with multiple data assimilation. Computers & Geosciences, 55:3–15, 2013

    2013

  41. [41]

    Training effective deep reinforcement learning agents for real-time life-cycle production optimization.Journal of Petroleum Science and Engineering, 208:109766, 2022

    Kai Zhang, Zhongzheng Wang, Guodong Chen, Liming Zhang, Yongfei Yang, Chuanjin Yao, Jian Wang, and Jun Yao. Training effective deep reinforcement learning agents for real-time life-cycle production optimization.Journal of Petroleum Science and Engineering, 208:109766, 2022

    2022

  42. [42]

    Deep reinforcement learning for optimal well control in subsurface systems with uncertain geology.Journal of Computational Physics, 477:111945, 2023

    Yusuf Nasir and Louis J Durlofsky. Deep reinforcement learning for optimal well control in subsurface systems with uncertain geology.Journal of Computational Physics, 477:111945, 2023

    2023

  43. [43]

    Stabilizing transformers for reinforcement learning

    Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. InInternational conference on machine learning, pages 7487–7498. PMLR, 2020

    2020

  44. [44]

    Well placement optimization with the covari- ance matrix adaptation evolution strategy and meta-models.Computational Geosciences, 16:75–92, 2012

    Zyed Bouzarkouna, Didier Yu Ding, and Anne Auger. Well placement optimization with the covari- ance matrix adaptation evolution strategy and meta-models.Computational Geosciences, 16:75–92, 2012

    2012

  45. [45]

    A derivative-free methodology with local and global search for the constrained joint optimization of well locations and controls

    Obiajulu J Isebor, Louis J Durlofsky, and David Echeverr´ ıa Ciaurri. A derivative-free methodology with local and global search for the constrained joint optimization of well locations and controls. Computational Geosciences, 18:463–482, 2014

    2014

  46. [46]

    Yusuf Nasir, Wei Yu, and Kamy Sepehrnoori. Hybrid derivative-free technique and effective machine learning surrogate for nonlinear constrained well placement and production optimization.Journal of Petroleum Science and Engineering, 186:106726, 2020

    2020

  47. [47]

    Application of a particle swarm optimization algorithm for determining optimum well location and type.Computational Geosciences, 14:183–198, 2010

    J´ erˆ ome E Onwunalu and Louis J Durlofsky. Application of a particle swarm optimization algorithm for determining optimum well location and type.Computational Geosciences, 14:183–198, 2010. 28

    2010

  48. [48]

    Optimal rate control under geologic uncertainty

    Ahmed H Alhuthali, Akhil Datta-Gupta, Bevan Yuen, and Jerry P Fontanilla. Optimal rate control under geologic uncertainty. InSPE Improved Oil Recovery Conference?, pages SPE–113628. SPE, 2008

    2008

  49. [49]

    Zhe Liu and Albert C Reynolds. A sequential-quadratic-programming-filter algorithm with a modified stochastic gradient for robust life-cycle optimization problems with nonlinear state constraints.SPE Journal, 25(04):1938–1963, 2020

    1938

  50. [50]

    Optimization of production operations in petroleum fields

    Pengju Wang, Michael Litvak, and Khalid Aziz. Optimization of production operations in petroleum fields. InSPE Annual Technical Conference and Exhibition?, pages SPE–77658. SPE, 2002

    2002

  51. [51]

    Ensemble-based multiobjective optimization of on/off control devices under geological uncertainty

    R-M-M Fonseca, Olwijn Leeuwenburgh, Ernesto Della Rossa, PM Van den Hof, and J-D-D Jansen. Ensemble-based multiobjective optimization of on/off control devices under geological uncertainty. SPE Reservoir Evaluation & Engineering, 18(04):554–563, 2015

    2015

  52. [52]

    Improving the ensemble-optimization method through covariance-matrix adaptation.Spe Journal, 20(01):155–168, 2015

    RM M Fonseca, Olwijn Leeuwenburgh, PMJ MJ Van den Hof, and JD D Jansen. Improving the ensemble-optimization method through covariance-matrix adaptation.Spe Journal, 20(01):155–168, 2015

    2015

  53. [53]

    Im- proved sampling strategies for ensemble-based optimization.Computational Geosciences, 24:1057– 1069, 2020

    KR Ramaswamy, RM Fonseca, Olwijn Leeuwenburgh, MM Siraj, and PMJ Van den Hof. Im- proved sampling strategies for ensemble-based optimization.Computational Geosciences, 24:1057– 1069, 2020

    2020

  54. [54]

    Joint optimization of oil well placement and controls.Computational Geosciences, 16:1061–1079, 2012

    Mathias C Bellout, David Echeverr´ ıa Ciaurri, Louis J Durlofsky, Bjarne Foss, and Jon Kleppe. Joint optimization of oil well placement and controls.Computational Geosciences, 16:1061–1079, 2012

    2012

  55. [55]

    Lianlin Li, Behnam Jafarpour, and M Reza Mohammad-Khaninezhad. A simultaneous perturba- tion stochastic approximation algorithm for coupled well placement and control optimization under geologic uncertainty.Computational Geosciences, 17:167–188, 2013

    2013

  56. [56]

    Joint optimization of number of wells, well locations and controls using a gradient-based algorithm.Chemical Engineering Research and Design, 92(7):1315– 1328, 2014

    Fahim Forouzanfar and Albert C Reynolds. Joint optimization of number of wells, well locations and controls using a gradient-based algorithm.Chemical Engineering Research and Design, 92(7):1315– 1328, 2014

    2014

  57. [57]

    A general method to select representative models for decision making and optimization under uncertainty.Computers & geosciences, 96:109–123, 2016

    Mehrdad G Shirangi and Louis J Durlofsky. A general method to select representative models for decision making and optimization under uncertainty.Computers & geosciences, 96:109–123, 2016

    2016

  58. [58]

    Closed-loop field development under uncertainty by use of optimization with sample validation.SPE Journal, 20(05):908–922, 2015

    Mehrdad G Shirangi and Louis J Durlofsky. Closed-loop field development under uncertainty by use of optimization with sample validation.SPE Journal, 20(05):908–922, 2015

    2015

  59. [59]

    Optimisa- tion of decision making under uncertainty throughout field lifetime: A fractured reservoir example

    Dan Arnold, Vasily Demyanov, Mike Christie, Alexander Bakay, and Konstantin Gopa. Optimisa- tion of decision making under uncertainty throughout field lifetime: A fractured reservoir example. Computers & Geosciences, 95:123–139, 2016

    2016

  60. [60]

    Reservoir development optimization under uncertainty for infill well placement in brownfield redevelopment.Journal of Petroleum Science and Engineering, 175:444–464, 2019

    Junko Hutahaean, Vasily Demyanov, and Mike Christie. Reservoir development optimization under uncertainty for infill well placement in brownfield redevelopment.Journal of Petroleum Science and Engineering, 175:444–464, 2019

    2019

  61. [61]

    Geophysical inversion with a neighbourhood algorithm—i

    Malcolm Sambridge. Geophysical inversion with a neighbourhood algorithm—i. searching a param- eter space.Geophysical journal international, 138(2):479–494, 1999

    1999

  62. [62]

    Geophysical inversion with a neighbourhood algorithm—ii

    Malcolm Sambridge. Geophysical inversion with a neighbourhood algorithm—ii. appraising the en- semble.Geophysical Journal International, 138(3):727–746, 1999

    1999

  63. [63]

    University of Cambridge, Department of Engineering Cambridge, UK, 1994

    Gavin A Rummery and Mahesan Niranjan.On-line Q-learning using connectionist systems, vol- ume 37. University of Cambridge, Department of Engineering Cambridge, UK, 1994. 29

    1994

  64. [64]

    Q-learning.Machine learning, 8:279–292, 1992

    Christopher JCH Watkins and Peter Dayan. Q-learning.Machine learning, 8:279–292, 1992

    1992

  65. [65]

    A reinforcement learning approach for waterflooding optimization in petroleum reservoirs.Engineering Applications of Artificial Intelligence, 77:98–116, 2019

    Farzad Hourfar, Hamed Jalaly Bidgoly, Behzad Moshiri, Karim Salahshoor, and Ali Elkamel. A reinforcement learning approach for waterflooding optimization in petroleum reservoirs.Engineering Applications of Artificial Intelligence, 77:98–116, 2019

    2019

  66. [66]

    Waterflooding optimization under geological uncertainties by using deep reinforcement learning algorithms

    Hongze Ma, Gaoming Yu, Yuehui She, and Yongan Gu. Waterflooding optimization under geological uncertainties by using deep reinforcement learning algorithms. InSPE Annual Technical Conference and Exhibition?, page D031S043R001. SPE, 2019

    2019

  67. [67]

    Deep reinforcement learning: reservoir optimization from pixels

    Ruslan Miftakhov, Abdulaziz Al-Qasim, and Igor Efremov. Deep reinforcement learning: reservoir optimization from pixels. InInternational Petroleum Technology Conference, page D021S052R002. IPTC, 2020

    2020

  68. [68]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  69. [69]

    Stochastic optimal well control in subsurface reservoirs using reinforcement learning.Engineering Applications of Artificial Intelligence, 114:105106, 2022

    Atish Dixit and Ahmed H ElSheikh. Stochastic optimal well control in subsurface reservoirs using reinforcement learning.Engineering Applications of Artificial Intelligence, 114:105106, 2022

    2022

  70. [70]

    Asynchronous Methods for Deep Reinforcement Learning

    Volodymyr Mnih. Asynchronous methods for deep reinforcement learning.arXiv preprint arXiv:1602.01783, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  71. [71]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. InInternational conference on machine learning, pages 1861–1870. PMLR, 2018

    2018

  72. [72]

    Deep reinforcement learning and adaptive policy transfer for generalizable well control optimization.Journal of Petroleum Science and Engineering, 217:110868, 2022

    Zhongzheng Wang, Kai Zhang, Jinding Zhang, Guodong Chen, Xiaopeng Ma, Guojing Xin, Jinzheng Kang, Hanjun Zhao, and Yongfei Yang. Deep reinforcement learning and adaptive policy transfer for generalizable well control optimization.Journal of Petroleum Science and Engineering, 217:110868, 2022

    2022

  73. [73]

    Evolutionary-assisted reinforcement learning for reservoir real-time production optimization under uncertainty.Petroleum Science, 20(1):261–276, 2023

    Zhong-Zheng Wang, Kai Zhang, Guo-Dong Chen, Jin-Ding Zhang, Wen-Dong Wang, Hao-Chen Wang, Li-Ming Zhang, Xia Yan, and Jun Yao. Evolutionary-assisted reinforcement learning for reservoir real-time production optimization under uncertainty.Petroleum Science, 20(1):261–276, 2023

    2023

  74. [74]

    Practical closed-loop reservoir management using deep reinforce- ment learning.SPE Journal, 28(03):1135–1148, 2023

    Yusuf Nasir and Louis J Durlofsky. Practical closed-loop reservoir management using deep reinforce- ment learning.SPE Journal, 28(03):1135–1148, 2023

    2023

  75. [75]

    Multi-asset closed-loop reservoir management using deep rein- forcement learning.Computational Geosciences, 28(1):23–42, 2024

    Yusuf Nasir and Louis J Durlofsky. Multi-asset closed-loop reservoir management using deep rein- forcement learning.Computational Geosciences, 28(1):23–42, 2024

    2024

  76. [76]

    Deep reinforcement learning for generalizable field development optimization.SPE Journal, 27(01):226–245, 2022

    Jincong He, Meng Tang, Chaoshun Hu, Shusei Tanaka, Kainan Wang, Xian-Huan Wen, and Yusuf Nasir. Deep reinforcement learning for generalizable field development optimization.SPE Journal, 27(01):226–245, 2022

    2022

  77. [77]

    Deep reinforcement learning for constrained field development optimization in subsurface two-phase flow

    Yusuf Nasir, Jincong He, Chaoshun Hu, Shusei Tanaka, Kainan Wang, and XianHuan Wen. Deep reinforcement learning for constrained field development optimization in subsurface two-phase flow. Frontiers in Applied Mathematics and Statistics, 7:689934, 2021. 30

    2021