Pith. sign in

REVIEW 2 major objections 34 references

AI weather models stay stable for a full year when they treat small spatio-temporal scales as noise to be damped rather than amplified.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-29 08:19 UTC pith:3IMDOLFP

load-bearing objection The paper delivers a usable three-regime taxonomy of long AI-weather rollouts from nine models plus ViT ablations, with the main empirical hook being that stable runs act as denoisers on small scales. the 2 major comments →

arxiv 2605.30184 v2 pith:3IMDOLFP submitted 2026-05-28 cs.LG physics.ao-ph

Can AI Weather Models Predict Beyond Two Weeks? A Quantitative Benchmark and Analysis of Long Rollouts

Fanny Lehmann , Firat Ozdemir , Yun Cheng , Torsten Hoefler , Sebastian Schemm , Benedikt Soja , Siddhartha Mishra This is my paper
classification cs.LG physics.ao-ph
keywords AI weather modelslong-term rolloutsmodel stabilityhigh-frequency energythree-regime taxonomyVision Transformerweather trajectoriesseasonality preservation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to classify why AI weather models break down beyond two weeks by running nine state-of-the-art models for an entire year and sorting the breakdowns into three regimes. A sympathetic reader would care because extending reliable forecasts past the current two-week limit would change how weather-dependent decisions are made at longer ranges. The central finding is that stability is controlled by how each model handles high-frequency energy at small scales: models that amplify it blow up or drift, while those that suppress it act like denoisers and produce distinct trajectories that stay tied to the initial conditions. Ablation experiments on Vision Transformer architectures confirm that specific design choices determine which regime appears.

Core claim

Year-long rollouts of nine AI weather models fall into three failure regimes—blow-up, drift, or loss of seasonality—whose occurrence is governed by the model's treatment of small spatio-temporal scales. Unstable models increase high-frequency energy, while stable models reduce it and thereby generate unique weather trajectories that remain conditioned on the starting state rather than converging to a generic attractor.

What carries the argument

The three-regime taxonomy (blow-up, drift, loss of seasonality) derived from energy spectra and seasonal cycle fidelity across the nine models, which directly links observed instability to amplification versus damping of high-frequency components.

Load-bearing premise

The failure modes seen in these nine models will remain the main ones that appear in future AI weather architectures.

What would settle it

A new model architecture that runs stably for a full year yet still increases high-frequency energy in its long rollouts, or one that damps high-frequency energy yet still loses seasonality or drifts.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Stable models produce deterministic yet non-repeating weather sequences that stay faithful to the initial atmospheric state over twelve months.
  • Architectural changes that make a model act as a denoiser on small scales eliminate blow-up and drift.
  • Adding controlled noise to model inputs can be used to test whether the model damps or amplifies high-frequency components.
  • The same three-regime classification applies across multiple current Vision Transformer designs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same denoising requirement may set limits on how far AI models can be pushed for seasonal-to-decadal climate runs without additional scale-aware constraints.
  • Benchmark suites for future models could routinely include a one-year rollout test plus a high-frequency energy spectrum check rather than stopping at 15 days.
  • If the taxonomy holds, developers could screen new architectures for stability by measuring only their response to injected small-scale noise instead of running full year-long simulations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The manuscript categorizes instabilities in long-horizon rollouts of AI weather models into three regimes (blow-up, drift, and loss of seasonality) based on year-long simulations of nine state-of-the-art models. It concludes that stability depends on the treatment of small spatio-temporal scales—unstable models amplify high-frequency energy while stable models act as denoisers—and that stable models produce unique, initial-state-conditioned trajectories rather than repeating training data. These findings are verified through ablation studies on Vision Transformer (ViT) architectures.

Significance. If the taxonomy and the causal link to small-scale treatment hold, the work supplies a concrete diagnostic framework and benchmark for extending AI weather forecasts beyond the current 15-day limit, an important practical gap. The direct numerical rollouts across multiple models plus the ViT ablations provide reproducible evidence for the scale-treatment hypothesis and the claim of unique trajectories, strengthening the contribution beyond anecdotal reports of instability.

major comments (2)
  1. [Abstract and Results (taxonomy definition)] The central taxonomy (blow-up, drift, loss of seasonality) is introduced in the abstract and presumably detailed in the Results section, yet no explicit quantitative thresholds, error bars, or decision criteria are stated for assigning a year-long trajectory to one regime versus another. Without these, the classification of the nine models remains partly subjective, directly affecting the load-bearing claim that stability is explained by small-scale treatment.
  2. [Ablation studies and Discussion] The claim that the three-regime taxonomy captures the dominant failure modes (and that stability hinges on small-scale handling) rests on rollouts of nine specific models plus ViT ablations. The manuscript does not test or bound whether new architectures employing different scale-separation mechanisms or training objectives could produce instabilities outside these regimes or uncorrelated with high-frequency amplification.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the detailed and constructive report. The two major comments highlight important points on the objectivity of the taxonomy and the generality of the findings. We address each below, agreeing where revisions are needed to strengthen the manuscript while defending the core claims on the basis of the presented evidence.

read point-by-point responses

  1. Referee: [Abstract and Results (taxonomy definition)] The central taxonomy (blow-up, drift, loss of seasonality) is introduced in the abstract and presumably detailed in the Results section, yet no explicit quantitative thresholds, error bars, or decision criteria are stated for assigning a year-long trajectory to one regime versus another. Without these, the classification of the nine models remains partly subjective, directly affecting the load-bearing claim that stability is explained by small-scale treatment.

    Authors: We agree that the taxonomy would benefit from more explicit quantitative criteria to reduce subjectivity. The regimes were assigned based on distinct observable behaviors across the year-long rollouts: blow-up by unbounded growth in kinetic energy and high-frequency spectral power; drift by systematic accumulation of mean-state bias without spectral blow-up; and loss of seasonality by decorrelation from the annual cycle in temperature and precipitation fields. In the revision we will add a dedicated subsection with precise decision criteria, including numerical thresholds on these metrics (e.g., high-frequency energy exceeding 3 standard deviations of the climatological spectrum for blow-up) and error bars computed from an ensemble of initial conditions. This will make the classification reproducible and directly support the link to small-scale treatment. revision: yes

  2. Referee: [Ablation studies and Discussion] The claim that the three-regime taxonomy captures the dominant failure modes (and that stability hinges on small-scale handling) rests on rollouts of nine specific models plus ViT ablations. The manuscript does not test or bound whether new architectures employing different scale-separation mechanisms or training objectives could produce instabilities outside these regimes or uncorrelated with high-frequency amplification.

    Authors: The taxonomy and causal link are grounded in the consistent behavior across nine diverse state-of-the-art models and the controlled ViT ablations that isolate the effect of small-scale filtering. The denoiser property observed in stable models provides a mechanistic explanation that is architecture-agnostic within the tested set. While we cannot exhaustively enumerate every conceivable future architecture, the empirical pattern and the spectral analysis offer a diagnostic framework that can be applied to new models. We will expand the Discussion to explicitly note this scope limitation and suggest the taxonomy as a starting point for future work rather than a universal bound. revision: partial

standing simulated objections not resolved
  • Whether entirely novel architectures with different scale-separation mechanisms could produce instabilities outside the three observed regimes or uncorrelated with high-frequency amplification cannot be bounded by any finite set of experiments.

Circularity Check

0 steps flagged

No circularity: empirical taxonomy from direct rollouts and ablations

full rationale

The paper's central claims rest on year-long numerical rollouts of nine existing AI weather models, followed by diagnostics of high-frequency energy and ablation studies on ViT architectures. No derivation chain, first-principles result, fitted parameter renamed as prediction, or self-citation load-bearing step is present; the three-regime taxonomy (blow-up, drift, loss of seasonality) is an empirical classification of observed behaviors rather than a reduction to inputs by construction. The analysis of stability via small-scale treatment is likewise grounded in the experimental outcomes, not in any self-referential definition or imported uniqueness theorem.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The abstract supplies no explicit free parameters, invented entities, or non-standard axioms; the work rests on the domain assumption that year-long unforced rollouts are a valid testbed for model stability.

axioms (1)
  • domain assumption Year-long unforced rollouts of AI weather models constitute a sufficient and representative test for identifying long-term instabilities.
    Implicit in the decision to run and categorize full-year trajectories without additional forcing or reinitialization.

pith-pipeline@v0.9.1-grok · 5704 in / 1262 out tokens · 32143 ms · 2026-06-29T08:19:41.904867+00:00 · methodology

0 comments
read the original abstract

While AI weather models excel at short-to-medium range forecasts (up to 15 days), they frequently suffer from ill-defined "instabilities" when rolled out over longer horizons. This work addresses the lack of a formal taxonomy by categorizing these failures into three distinct regimes: blow-up, drift, and loss of seasonality, through year-long rollouts of nine state-of-the-art AI weather models. Our analysis reveals that stability hinges on the treatment of small spatio-temporal scales: unstable models amplify high-frequency energy, while stable models act as denoisers when noise is added to their inputs. Far from reducing these models to mere stochastic parrots, our findings highlight that stable models generate unique weather trajectories, conditioned on the initial state. We verify our findings through ablation studies on architectural design choices, conducted using state-of-the-art Vision Transformer (ViT) AI weather model architectures.

Figures

Figures reproduced from arXiv: 2605.30184 by Benedikt Soja, Fanny Lehmann, Firat Ozdemir, Sebastian Schemm, Siddhartha Mishra, Torsten Hoefler, Yun Cheng.

Figure 1
Figure 1. Figure 1: Rollouts initialized on January 1st, 2021 for nine AI weather models. The lines show 2-meter temperature [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Error between the reference rollout and the rollout initialised with a noisy input (additive noise with [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: 2-meter temperature in East Asia for Aurora and AIFS initialized on January 1st, 2021 (left column), AIFS [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: 4-year rollout of 2-meter temperature for a) four small Aurora models trained from scratch with different [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: (left) RMSE between the monthly seasonal cycle of the Aurora [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: QQ plots comparing ERA5 and the model tail quantiles of regional spatial maximum (hot, top) and minimum [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 1 canonical work pages

  1. [1]

    Armegioiu

    V . Armegioiu. Memory-Conditioned Flow-Matching for Stable Autoregressive PDE Rollouts, Feb. 2026

    2026

  2. [2]

    K. Bi, L. Xie, H. Zhang, X. Chen, X. Gu, and Q. Tian. Accurate medium-range global weather forecasting with 3D neural networks.Nature, 619(7970):533–538, July 2023. Pangu

    2023

  3. [3]

    Bodnar, W

    C. Bodnar, W. P. Bruinsma, A. Lucic, M. Stanley, A. Allen, J. Brandstetter, P. Garvan, M. Riechert, J. A. Weyn, H. Dong, J. K. Gupta, K. Thambiratnam, A. T. Archibald, C.-C. Wu, E. Heider, M. Welling, R. E. Turner, and P. Perdikaris. A foundation model for the Earth system.Nature, May 2025

    2025

  4. [4]

    Bonavita

    M. Bonavita. On Some Limitations of Current Machine Learning Weather Prediction Models.Geophysical Research Letters, 51(12):e2023GL107377, June 2024

    2024

  5. [5]

    Bonev, T

    B. Bonev, T. Kurth, C. Hundt, J. Pathak, M. Baust, K. Kashinath, and A. Anandkumar. Spherical Fourier neural operators: Learning stable dynamics on the sphere. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors,Proceedings of the 40th international conference on machine learning, volume 202 of Proceedings of machine lea...

    2023

  6. [6]

    arXiv preprint arXiv:2505.17638 , year=

    T. Bonnaire, R. Urfin, G. Biroli, and M. Mézard. Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training, Oct. 2025. arXiv:2505.17638 [cs]

    work page arXiv 2025

  7. [7]

    Brandstetter, D

    J. Brandstetter, D. Worrall, and M. Welling. Message Passing Neural PDE Solvers. 2022

    2022

  8. [8]

    Chattopadhyay, Y

    A. Chattopadhyay, Y . Q. Sun, and P. Hassanzadeh. Challenges of learning multi-scale dynamics with AI weather models: Implications for stability and one solution, Dec. 2024. long rollouts

    2024

  9. [9]

    L. Chen, X. Zhong, F. Zhang, Y . Cheng, Y . Xu, Y . Qi, and H. Li. FuXi: a cascade machine learning forecasting system for 15-day global weather forecast.npj Climate and Atmospheric Science, 6(1):190, Nov. 2023

    2023

  10. [10]

    Couairon, C

    G. Couairon, C. Lessig, A. Charantonis, and C. Monteleoni. ArchesWeather: An efficient AI weather forecasting model at 1.5{\deg} resolution, May 2024

    2024

  11. [11]

    Cresswell-Clay, B

    N. Cresswell-Clay, B. Liu, D. R. Durran, Z. Liu, Z. I. Espinosa, R. A. Moreno, and M. Karlbauer. A Deep Learning Earth System Model for Efficient Simulation of the Observed Climate.AGU Advances, 6(4):e2025A V001706, Aug. 2025

    2025

  12. [12]

    Gallusser, S

    F. Gallusser, S. Hentschel, A. Krause, and A. Hotho. Exploring Design Choices for Autoregressive Deep Learning Climate Models, May 2025. long rollouts

    2025

  13. [13]

    X. Gu, C. Du, T. Pang, C. Li, M. Lin, and Y . Wang. On Memorization in Diffusion Models, 2023. Version Number: 2

    2023

  14. [14]

    H. Guan, T. Arcomano, A. Chattopadhyay, and R. Maulik. LUCIE: A Lightweight Uncoupled ClImate Emulator with long-term stability and physical consistency for O(1000)-member ensembles, 2024

    2024

  15. [15]

    Hersbach, B

    H. Hersbach, B. Bell, P. Berrisford, S. Hirahara, A. Horányi, J. Muñoz-Sabater, J. Nicolas, C. Peubey, R. Radu, D. Schepers, A. Simmons, C. Soci, S. Abdalla, X. Abellan, G. Balsamo, P. Bechtold, G. Biavati, J. Bidlot, M. Bonavita, G. De Chiara, P. Dahlgren, D. Dee, M. Diamantakis, R. Dragani, J. Flemming, R. Forbes, M. Fuentes, A. Geer, L. Haimberger, S. ...

    1999

  16. [16]

    Karlbauer, N

    M. Karlbauer, N. Cresswell-Clay, D. R. Durran, R. A. Moreno, T. Kurth, B. Bonev, N. Brenowitz, and M. V . Butz. Advancing Parsimonious Deep Learning Weather Prediction Using the HEALPix Mesh.Journal of Advances in Modeling Earth Systems, 16(8):e2023MS004021, Aug. 2024

    2024

  17. [17]

    Kochkov, J

    D. Kochkov, J. Yuval, I. Langmore, P. Norgaard, J. Smith, G. Mooers, M. Klöwer, J. Lottes, S. Rasp, P. Düben, S. Hatfield, P. Battaglia, A. Sanchez-Gonzalez, M. Willson, M. P. Brenner, and S. Hoyer. Neural general circulation models for weather and climate.Nature, 632(8027):1060–1066, Aug. 2024

    2024

  18. [18]

    R. Lam, A. Sanchez-Gonzalez, M. Willson, P. Wirnsberger, M. Fortunato, F. Alet, S. Ravuri, T. Ewalds, Z. Eaton- Rosen, W. Hu, A. Merose, S. Hoyer, G. Holland, O. Vinyals, J. Stott, A. Pritzel, S. Mohamed, and P. Battaglia. Learning skillful medium-range global weather forecasting.Science, page eadi2336, Nov. 2023

    2023

  19. [19]

    S. Lang, M. Alexe, M. Chantry, J. Dramsch, F. Pinault, B. Raoult, M. C. A. Clare, C. Lessig, M. Maier-Gerber, L. Magnusson, Z. B. Bouallègue, A. P. Nemesio, P. D. Dueben, A. Brown, F. Pappenberger, and F. Rabier. AIFS - ECMWF’s data-driven forecasting system, June 2024

    2024

  20. [20]

    S. Lang, M. Alexe, M. C. A. Clare, C. Roberts, R. Adewoyin, Z. B. Bouallègue, M. Chantry, J. Dramsch, P. D. Dueben, S. Hahner, P. Maciel, A. Prieto-Nemesio, C. O’Brien, F. Pinault, J. Polster, B. Raoult, S. Tietsche, and 11 M. Leutbecher. AIFS-CRPS: Ensemble forecasting using a model trained with a loss function based on the Continuous Ranked Probability ...

    2024

  21. [21]

    Lippe, B

    P. Lippe, B. Veeling, P. Perdikaris, R. Turner, and J. Brandstetter. PDE-refiner: Achieving accurate long rollouts with neural PDE solvers. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in neural information processing systems, volume 36, pages 67398–67433. Curran Associates, Inc., 2023

    2023

  22. [22]

    Loegel, J

    O. Loegel, J. Talib, F. Vitart, J. Hoffmann, and M. Chantry. The AI Weather Quest: an international competition for sub-seasonal forecasting with AI.Machine Learning: Earth, 1(1):010701, Dec. 2025

    2025

  23. [23]

    McCabe, P

    M. McCabe, P. Harrington, S. Subramanian, and J. Brown. Towards stability of autoregressive neural operators. Transactions on Machine Learning Research, 2023

    2023

  24. [24]

    Nie and Y

    F. Nie and Y . Suzuki. JAWS: Enhancing Long-term Rollout of Neural PDE Solvers via Spatially-Adaptive Jacobian Regularization, 2026

    2026

  25. [25]

    Pathak, S

    J. Pathak, S. Subramanian, P. Harrington, S. Raja, A. Chattopadhyay, M. Mardani, T. Kurth, D. Hall, Z. Li, K. Azizzadenesheli, P. Hassanzadeh, K. Kashinath, and A. Anandkumar. FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators, Feb. 2022

    2022

  26. [26]

    Pedersen, L

    C. Pedersen, L. Zanna, and J. Bruna. Thermalizer: Stable autoregressive neural emulation of spatiotemporal chaos, Mar. 2025. long rollout

    2025

  27. [27]

    Price, A

    I. Price, A. Sanchez-Gonzalez, F. Alet, T. R. Andersson, A. El-Kadi, D. Masters, T. Ewalds, J. Stott, S. Mohamed, P. Battaglia, R. Lam, and M. Willson. Probabilistic weather forecasting with machine learning.Nature, Dec. 2024. GenCast

    2024

  28. [28]

    S. Rasp, S. Hoyer, A. Merose, I. Langmore, P. Battaglia, T. Russel, A. Sanchez-Gonzalez, V . Yang, R. Carver, S. Agrawal, M. Chantry, Z. B. Bouallegue, P. Dueben, C. Bromberg, J. Sisk, L. Barrington, A. Bell, and F. Sha. WeatherBench 2: A benchmark for the next generation of data-driven global weather models, Jan. 2024

    2024

  29. [29]

    Watt-Meyer, B

    O. Watt-Meyer, B. Henn, J. McGibbon, S. K. Clark, A. Kwa, W. A. Perkins, E. Wu, L. Harris, and C. S. Bretherton. ACE2: Accurately learning subseasonal to decadal atmospheric variability and forced responses, Nov. 2024

    2024

  30. [30]

    J. D. Willard, P. Harrington, S. Subramanian, A. Mahesh, T. A. O’Brien, and W. D. Collins. Analyzing and Exploring Training Recipes for Large-Scale Transformer-Based Weather Prediction.Artificial Intelligence for the Earth Systems, 4(2):240061, Apr. 2025

    2025

  31. [31]

    H. Wu, Y . Gao, R. Shu, K. Wang, R. Gou, C. Wu, X. Liu, J. He, S. Cao, J. Fang, X. Shi, F. Tao, Q. Song, S. Ji, Y . Xiang, Y . Sun, J. Li, F. Xu, H. Dong, H. Wang, F. Zhang, P. Zhao, X. Wu, Q. Wen, D. Chen, and X. Huang. Advanced long-term earth system forecasting by learning the small-scale nature, 2025

    2025

  32. [32]

    W. Xu, K. Chen, T. Han, H. Chen, W. Ouyang, and L. Bai. ExtremeCast: Boosting Extreme Value Prediction for Global Weather Forecast. 2024. extreme values

    2024

  33. [33]

    T. Yoon, J. Y . Choi, S. Kwon, and E. K. Ryu. Diffusion probabilistic models generalize when they fail to memorize. InICML 2023 workshop on structured probabilistic inference & generative modeling, 2023

    2023

  34. [34]

    Zhou and S

    H. Zhou and S. Cheng. Improving long-term autoregressive spatiotemporal predictions: A proof of concept with fluid dynamics.Computer Methods in Applied Mechanics and Engineering, 447:118332, Dec. 2025. 12 A Weather and Climate Models Table A.1: Non exhaustive summary of state-of-the-art AI weather and climate models referenced in this work. Spatial resolu...

    2025