Pith. sign in

REVIEW 2 major objections 2 minor 3 references

Conditioning weather-trained ML models on monthly SST and SIC produces stable multi-decadal climate simulations matching ERA5.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-28 23:45 UTC pith:P6NI6L5K

load-bearing objection These weather models stay stable over decades with monthly SST/SIC forcing and reproduce ERA5 features under AIMIP, but the strength of that match rests on details not visible in the summary. the 2 major comments →

arxiv 2605.29976 v1 pith:P6NI6L5K submitted 2026-05-28 physics.ao-ph cs.AI

Evaluating Skill and Stability of ArchesWeather and ArchesWeatherGen under Multi-Decadal Climate Simulations

Renu Singh , Robert Brunstein , Antonia Jost , Thomas Rackow , Claire Monteleoni , Yana Hasson , Christian Lessig , Guillaume Couairon This is my paper
classification physics.ao-ph cs.AI
keywords machine learningclimate simulationArchesWeatherSST conditioningAIMIPERA5 evaluationatmospheric modelingmulti-decadal stability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether two machine learning models originally built for short-term weather prediction can be turned into usable tools for long-term climate work. It adds monthly mean sea surface temperature and sea ice cover as boundary conditions and runs them following the AIMIP experimental protocol. The central result is that the adapted models stay stable over decades, keep a consistent annual cycle, and reproduce the observed climatology, large-scale flows, year-to-year changes, and extremes from ERA5 reanalysis. A reader would care because this shows a low-cost way to repurpose existing weather models for climate-length runs instead of building new ones from scratch.

Core claim

Despite being originally developed for weather forecasting, forced configurations of ArchesWeather and ArchesWeatherGen produce stable long-term climate simulations, have a stable annual cycle, and capture the drift of many climate variables. The models faithfully reproduce ERA5's climatology, large-scale circulations and interannual variability, and they capture the tails of the distributions.

What carries the argument

Monthly mean SST and SIC conditioning as boundary conditions to adapt weather models into forced atmospheric climate models under the AIMIP protocol.

Load-bearing premise

Monthly mean SST and SIC conditioning alone is enough to turn weather models into stable forced climate models for multi-decadal runs.

What would settle it

A multi-decadal AIMIP run that shows growing unbounded drift in global temperature, circulation strength, or annual cycle amplitude beyond ERA5 levels would falsify the stability claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript evaluates the adaptation of ArchesWeather (deterministic) and ArchesWeatherGen (probabilistic flow-matching) weather-forecasting models to multi-decadal climate simulation. By adding conditioning on monthly-mean SST and SIC as boundary conditions and following the AIMIP Phase 1 protocol, the authors claim that the forced configurations produce stable long-term runs with a stable annual cycle, capture drifts in many climate variables, faithfully reproduce ERA5 climatology, large-scale circulations, interannual variability and distribution tails, and outperform or match numerical climate models in key respects; the evaluation includes forced/unforced comparisons and ablation studies on design choices.

Significance. If the quantitative results hold, the work would establish that weather-trained ML models can be repurposed for stable forced atmospheric climate simulation with only monthly SST/SIC conditioning, offering a potentially efficient route to ensemble climate runs and uncertainty quantification that complements traditional GCMs.

major comments (2)
  1. [Methods / AIMIP protocol section] The central claim that monthly-mean SST/SIC conditioning alone suffices for drift-free multi-decadal stability (abstract and AIMIP protocol description) is load-bearing, yet the manuscript provides no explicit quantification of drift rates (e.g., linear trends in global-mean temperature or zonal wind with confidence intervals) or a clear statement of the exact conditioning implementation (how the monthly fields are upsampled and injected into the network).
  2. [Ablation studies and results sections] Ablation results on the conditioning and forced vs. unforced configurations are referenced but lack tabulated metrics (e.g., RMSE or bias for key variables across periods) with error bars or statistical significance tests, making it impossible to judge whether the reported stability and ERA5 fidelity are robust or sensitive to post-hoc choices.
minor comments (2)
  1. [Figures] Figure captions and axis labels should explicitly state the time period, ensemble size, and variable units for all climatology and variability plots to allow direct comparison with ERA5 and CMIP-style benchmarks.
  2. [Abstract] The abstract states that the models 'capture the tails of the distributions' but does not define the tail metric (e.g., 99th percentile bias or extreme-event frequency); this should be clarified with a precise definition and reference to the corresponding figure or table.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments and the recommendation for major revision. We will update the manuscript to provide the requested quantifications and clarifications, as detailed in our point-by-point responses below.

read point-by-point responses

  1. Referee: [Methods / AIMIP protocol section] The central claim that monthly-mean SST/SIC conditioning alone suffices for drift-free multi-decadal stability (abstract and AIMIP protocol description) is load-bearing, yet the manuscript provides no explicit quantification of drift rates (e.g., linear trends in global-mean temperature or zonal wind with confidence intervals) or a clear statement of the exact conditioning implementation (how the monthly fields are upsampled and injected into the network).

    Authors: We agree that providing explicit quantification of drift rates would strengthen the central claim. In the revised manuscript, we will add analyses of linear trends in global-mean temperature, zonal wind, and other key variables, including confidence intervals. We will also clarify the conditioning implementation in the Methods section by detailing how monthly SST and SIC fields are upsampled (e.g., via interpolation) and injected into the network architecture. These details were part of our experimental setup but not fully elaborated. revision: yes

  2. Referee: [Ablation studies and results sections] Ablation results on the conditioning and forced vs. unforced configurations are referenced but lack tabulated metrics (e.g., RMSE or bias for key variables across periods) with error bars or statistical significance tests, making it impossible to judge whether the reported stability and ERA5 fidelity are robust or sensitive to post-hoc choices.

    Authors: We acknowledge the need for more rigorous presentation of the ablation results. We will include tabulated metrics in the revised version, reporting RMSE, biases, and other relevant statistics for key variables across different time periods. These tables will incorporate error bars (e.g., standard deviations across ensemble members or runs) and statistical significance tests (e.g., t-tests or similar) to assess robustness. This will allow better evaluation of the sensitivity to design choices. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper performs empirical evaluation of adapted ML weather models under the external AIMIP protocol, comparing outputs directly to independent ERA5 reanalysis and numerical climate models. No equations, derivations, or 'predictions' are presented that reduce by construction to fitted inputs or self-citations. The conditioning on monthly SST/SIC is an explicit adaptation step whose stability is tested against external benchmarks rather than assumed or renamed from prior results. All load-bearing claims rest on falsifiable comparisons outside the paper's own fitted values.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The paper is an empirical evaluation study; the abstract introduces no new free parameters, no ad-hoc axioms beyond standard climate modeling practices, and no invented entities.

axioms (1)
  • domain assumption The AIMIP Phase 1 protocol provides a valid standardized experimental setup for evaluating ML-based forced atmospheric models analogous to AMIP.
    Invoked when stating that the evaluation follows the AIMIP protocol.

pith-pipeline@v0.9.1-grok · 5804 in / 1341 out tokens · 33506 ms · 2026-06-28T23:45:47.915830+00:00 · methodology

0 comments
read the original abstract

We evaluate the climate simulation capabilities of ArchesWeather and ArchesWeatherGen, two machine learning models originally trained for weather forecasting and evaluated up to a 10-day lead time. ArchesWeather is a deterministic model, while ArchesWeatherGen is a probabilistic flow-matching model leveraging ArchesWeather's forecasts, enabling ensemble-based uncertainty quantification. In this work, we adapt these models to act as forced atmospheric models by using additional conditioning on the monthly mean sea surface temperature (SST) and sea ice cover (SIC) as boundary conditions. In particular, we follow the AI Model Intercomparison Project (AIMIP) Phase 1 protocol, which, analogous to the Atmospheric Model Intercomparison Project (AMIP), proposes a standardized experimental setup to evaluate the climate skill of ML-based forced atmospheric models. We present a comprehensive evaluation of both models under these conditions, including comparison against numerical climate models, ablation studies that examine key design choices in the extension, and an analysis of forced versus unforced configurations. Despite being originally developed for weather forecasting, we demonstrate that forced configurations of ArchesWeather and ArchesWeatherGen produce stable long-term climate simulations, have a stable annual cycle, and capture the drift of many climate variables. The models faithfully reproduce ERA5's climatology, large-scale circulations and interannual variability, and they capture the tails of the distributions.

Figures

Figures reproduced from arXiv: 2605.29976 by Antonia Jost, Christian Lessig, Claire Monteleoni, Guillaume Couairon, Renu Singh, Robert Brunstein, Thomas Rackow, Yana Hasson.

Figure 1
Figure 1. Figure 1: Global annual means of surface air temperature for ArchesWeatherGen, ArchesWeather [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: RMSE of monthly climatology for train (last 10 years only) and the test period. In [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Bias maps of the mean climate for temperature at 850 hPa (left), eastward wind at 850 [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Cram´er–von Mises distances show that overall, the final ArchesWeatherGen configuration, [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Histograms showcasing the distribution of eastward wind at 850 hPa, specific humidity at [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Spherical harmonics spectra of our models compared against ERA5 and MPI-ESM2- [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Leading empirical orthogonal functions of the northern hemisphere showing the tripole [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Leading empirical orthogonal functions of the southern hemisphere. All models show the [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Taylor diagrams for the annular modes. For the SAM, ArchesWeatherGen and Arch [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: a) The computed Southern Oscillation Index for ArchesWeatherGen. In b), the power [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: The Webster-Yang monsoon index a) computed for ArchesWeather-Mx4, ArchesWeath [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Return periods for surface air temperature and eastward wind at 850 hPa. ArchesWeath [PITH_FULL_IMAGE:figures/full_fig_p019_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Global annual mean predictions for surface air temperature from ArchesWeatherGen [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Spatial bias maps showing the global response of our model for surface air temperature, [PITH_FULL_IMAGE:figures/full_fig_p021_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Latitudinal profiles for surface air temperature, eastward wind at 850 hPa and specific [PITH_FULL_IMAGE:figures/full_fig_p021_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Global annual means of surface air temperature for ArchesWeatherGen, ArchesWeather [PITH_FULL_IMAGE:figures/full_fig_p026_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Power spectral densities of our models compared against ERA5. While the forced [PITH_FULL_IMAGE:figures/full_fig_p027_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: RMSE of monthly climatology for test period (2015-2024). The bar magnitude is the [PITH_FULL_IMAGE:figures/full_fig_p029_18.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

3 extracted references · 3 canonical work pages

  1. [1]

    Spencer K

    URLhttps://arxiv.org/abs/2505.06474. Spencer K. Clark, Oliver Watt-Meyer, Anna Kwa, Jeremy McGibbon, Brian Henn, W. Andre Perkins, Elynn Wu, Lucas M. Harris, and Christopher S. Bretherton. Ace2-som: Coupling an ml atmospheric emulator to a slab ocean and learning the sensitivity of climate to changed co 2,

    work page arXiv

  2. [2]

    Guillaume Couairon, Renu Singh, Anastase Charantonis, Christian Lessig, and Claire Monteleoni

    URLhttps://arxiv.org/abs/2412.04418. Guillaume Couairon, Renu Singh, Anastase Charantonis, Christian Lessig, and Claire Monteleoni. Archesweathergen: Skillful and compute-efficient probabilistic weather forecasting with machine learning.Science Advances, 12(17):eadx2372, 2026. 23 Nathaniel Cresswell-Clay, Bowen Liu, Dale R. Durran, Zihui Liu, Zachary I. E...

    work page doi:10.1175/1520-0442(2003)16 2026

  3. [3]

    URLhttps://rmets.onlinelibrary.wiley.com/doi/ abs/10.1002/joc.1499

    https://doi.org/10.1002/joc.1499. URLhttps://rmets.onlinelibrary.wiley.com/doi/ abs/10.1002/joc.1499. Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, Andr´ as Hor´ anyi, Joaqu´ ın Mu˜ noz-Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, et al. The era5 global reanalysis. Quarterly journal of the royal meteorological society...

    work page doi:10.1002/joc.1499 1999