pith. sign in

arxiv: 2605.26484 · v1 · pith:CND7DI2Dnew · submitted 2026-05-26 · 💻 cs.LG

Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training

Wenjie Zhou , Bohan Wang , Hongtao Zhang , Chenxi Jia , Wei Chen , Xueqi Cheng This is my paper

Pith reviewed 2026-06-29 19:45 UTC · model grok-4.3

classification 💻 cs.LG
keywords model mergingrank-1 subspacepre-training trajectorieslanguage modelsextrapolationloss landscapetraining-free optimization
0
0 comments X

The pith

Merged checkpoints in LLM pre-training collapse onto a stable one-dimensional subspace allowing loss reduction by extrapolation alone.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The authors observe that averaging consecutive checkpoints during late pre-training produces points that lie on an approximately one-dimensional linear manifold, unlike the wild oscillations of individual optimization steps. They attribute this to the loss landscape resembling a river valley, where averaging smooths away high-curvature noise and exposes the main direction of descent. From this they derive Extra-Merge, a method that moves beyond the last merged point along this line to reach lower loss without running any new training steps. Tests on GPT-2 and LLaMA models up to 2 billion parameters show this approach beats ordinary merging on downstream zero-shot accuracy.

Core claim

While raw optimization steps oscillate violently, consecutive merged checkpoints collapse onto a stable, approximately one-dimensional linear manifold. This is grounded in a river-valley landscape analysis where averaging acts as a geometric low-pass filter that dampens high-curvature noise to reveal the optimal descent direction. Extra-Merge extrapolates along this subspace to minimize loss without additional gradient updates.

What carries the argument

The Rank-1 Subspace formed by consecutive merged checkpoints, which is used as the direction for training-free extrapolation in the Extra-Merge method.

If this is right

  • Extra-Merge consistently outperforms standard merging baselines across GPT-2 and LLaMA families from 124M to 2B parameters.
  • It produces consistent zero-shot accuracy gains on Pythia-12B downstream tasks.
  • The approach generalizes to the Muon optimizer.
  • Loss can be minimized along the subspace without any gradient updates or additional training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This suggests that late-stage pre-training trajectories contain hidden low-dimensional structure once noise is filtered.
  • Similar merging and extrapolation could be tested on other architectures or training regimes to see if the rank-1 property holds.
  • Combining Extra-Merge with other post-training techniques might produce further improvements in efficiency.
  • The river-valley interpretation could motivate new smoothing methods for analyzing optimization paths.

Load-bearing premise

That the averaging process in merging removes high-curvature noise to expose a linear optimal descent direction in the loss landscape.

What would settle it

Observing that points obtained by extrapolating along the line between two consecutive merged checkpoints have higher loss than the merged points themselves would falsify the utility of the subspace for improvement.

Figures

Figures reproduced from arXiv: 2605.26484 by Bohan Wang, Chenxi Jia, Hongtao Zhang, Wei Chen, Wenjie Zhou, Xueqi Cheng.

Figure 1
Figure 1. Figure 1: Comparison of merging strategies. We compare the validation loss of the raw optimization trajectory against PMA (Uniform) and EMA-based merging schemes (γ ∈ {0.1, 0.2}). Experiments are conducted on GPT-2 Small (124M) trained on FineWeb (τ = 500, n = 8). where w n−1 i=0 are scalar weights determining the contribution of each checkpoint. This formulation allows for various weighting schemes, such as Exponen… view at source ↗
Figure 2
Figure 2. Figure 2: Geometric shift from Convexity to Monotonicity. We visualize the loss landscape along the linear interpolation path θ(α) = (1 − α)θt + αθt+τ between consecutive checkpoints (τ = 500). Left (Raw): The trajectory exhibits a convex basin profile, where the midpoint loss is significantly lower than the endpoints. Right (Merged): The landscape transforms into a monotonic descent, under the merging strategy of (… view at source ↗
Figure 3
Figure 3. Figure 3: PCA analysis of raw and merged trajectories. We perform PCA on sequences of K = 5 consecutive checkpoints for GPT-2 and LLaMA models (From 124M–2B). Top (Spectral Concentration): Merged trajectories (pink) exhibit a dominant Rank-1 structure, with the first principal component (PC1) explaining > 94% of the variance. In contrast, raw trajectories (blue) disperse energy across multiple components. Bottom (Tr… view at source ↗
Figure 4
Figure 4. Figure 4: Robustness of the Rank-1 Subspace. We analyze the persistence of linearity (PC1 Explained Variance Ratio) as a function of the trajectory length K (number of checkpoints included in PCA), varying the merging window size n in GPT-2 Small experiment. Merged trajectories with sufficient window sizes (n > 4) maintain a robust linear structure (EV R > 0.8) even over long horizons in the late-stage of training (… view at source ↗
Figure 5
Figure 5. Figure 5: (a) A visual illustration of the loss landscape. While raw optimization steps (blue trajectory) oscillate violently across the high-curvature valley walls, the merged checkpoints (pink nodes) filter out these oscillations, collapsing onto the stable valley floor (the Rank-1 Subspace). Extra-Merge (dashed arrow) identifies the tangent of this subspace to extrapolate towards the ”undiscovered” low-loss regio… view at source ↗
Figure 6
Figure 6. Figure 6: Validation loss trajectories across model scales and schedules. We compare Raw, PMA , and Extra-Merge on (First row) GPT-2 family trained with WSD scheduler, and (Last row) LLaMA family trained with Cosine scheduler. While PMA gradually aligns with the raw baseline as the learning rate decays, Extra-Merge achieve consistently lower loss [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Validation loss trajectory on FineWeb using the Muon optimizer. We compare the Raw baseline, PMA, and Extra-Merge on GPT-2 Small. Extra-Merge consistently outperforms PMA. We follow the experiment setting of the standard modded nano-gpt configuration. We maintain the identical merg￾ing protocol used in the AdamW experiments to ensure consistency. Detailed hyperparameters and experimental settings are provi… view at source ↗
read the original abstract

Model merging has emerged as a lightweight paradigm for enhancing Large Language Models (LLMs), yet its underlying mechanisms remain poorly understood. In this work, we analyze late-stage pre-training trajectories and uncover a \textbf{Rank-1 Subspace} phenomenon: while raw optimization steps oscillate violently, consecutive \emph{merged} checkpoints collapse onto a stable, approximately one-dimensional linear manifold. We theoretically ground this observation in a \emph{river-valley} landscape analysis: averaging acts as a geometric low-pass filter that dampens high-curvature noise to reveal the optimal descent direction. Capitalizing on this insight, we propose \textbf{Extra-Merge}, a training-free strategy that extrapolates along this subspace to minimize loss without additional gradient updates. Extensive experiments across GPT-2 and LLaMA families (124M to 2B) demonstrate that Extra-Merge consistently outperforms standard merging baselines. Notably, it yields consistent zero-shot accuracy gains on Pythia-12B downstream tasks and generalizes effectively to the Muon optimizer \citep{jordan2024muon}.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper claims that during late-stage pre-training of LLMs, raw optimization trajectories oscillate while consecutive merged checkpoints collapse onto a stable, approximately rank-1 linear manifold. This is theoretically grounded in a river-valley landscape analysis in which averaging acts as a geometric low-pass filter that reveals the optimal descent direction. The authors introduce Extra-Merge, a training-free extrapolation procedure along this subspace that reduces loss without gradient steps, and report consistent gains over standard merging baselines on GPT-2 and LLaMA models (124M–2B parameters), with additional results on Pythia-12B zero-shot tasks and the Muon optimizer.

Significance. If the rank-1 manifold observation and the effectiveness of the extrapolation hold under rigorous verification, the work would supply both a geometric explanation for merging behavior and a practical, training-free improvement technique. The breadth of experiments across model families and the reported generalization to Muon constitute concrete strengths that would increase the result’s potential influence on efficient LLM post-training methods.

major comments (2)
  1. [theoretical analysis / abstract] River-valley landscape analysis (abstract and theoretical section): the claim that averaging necessarily isolates a single stable rank-1 direction as the optimal descent direction is asserted without an explicit projection operator, eigenvalue bound, or extrapolation-error term. This step is load-bearing for the justification of Extra-Merge; its absence leaves the central geometric argument without a verifiable guarantee that loss decreases along the extrapolated vector.
  2. [experiments] Experimental results (presumed §4–5): no error bars, ablation on the extrapolation coefficient, or dataset descriptions are supplied in the abstract-level summary, rendering it impossible to judge whether the reported zero-shot gains are statistically reliable or sensitive to hyper-parameter choices. This directly affects the empirical support for the rank-1 subspace claim.
minor comments (2)
  1. [method] Notation for the merged checkpoint sequence and the extrapolation vector should be introduced with explicit equations rather than prose descriptions.
  2. [results] The statement that the manifold is “approximately one-dimensional” would benefit from a quantitative measure (e.g., explained variance ratio of the first principal component) reported in a table.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on the theoretical analysis and experimental reporting. We address each major comment below and will incorporate revisions to strengthen the manuscript.

read point-by-point responses

  1. Referee: River-valley landscape analysis (abstract and theoretical section): the claim that averaging necessarily isolates a single stable rank-1 direction as the optimal descent direction is asserted without an explicit projection operator, eigenvalue bound, or extrapolation-error term. This step is load-bearing for the justification of Extra-Merge; its absence leaves the central geometric argument without a verifiable guarantee that loss decreases along the extrapolated vector.

    Authors: We agree that the river-valley analysis would benefit from greater formality. In the revised manuscript we will introduce an explicit orthogonal projection operator onto the estimated rank-1 subspace, derive a simple eigenvalue bound on the curvature along the valley floor, and add an extrapolation-error term that bounds the loss change under the low-pass filtering interpretation. These additions will appear in the theoretical section and will directly support the loss-decrease claim for Extra-Merge. revision: yes

  2. Referee: Experimental results (presumed §4–5): no error bars, ablation on the extrapolation coefficient, or dataset descriptions are supplied in the abstract-level summary, rendering it impossible to judge whether the reported zero-shot gains are statistically reliable or sensitive to hyper-parameter choices. This directly affects the empirical support for the rank-1 subspace claim.

    Authors: Dataset descriptions for both pre-training corpora and downstream zero-shot tasks are already present in Sections 4 and 5. However, we acknowledge that error bars across random seeds and a systematic ablation on the extrapolation coefficient are missing. We will add multi-seed error bars to all reported metrics and include an ablation varying the extrapolation coefficient (with corresponding loss and accuracy curves) in the revised experimental section. revision: yes

Circularity Check

0 steps flagged

No circularity; empirical observation and river-valley analysis remain independent

full rationale

The paper reports an empirical collapse of merged checkpoints onto a rank-1 manifold, then supplies a separate river-valley landscape argument that averaging functions as a geometric low-pass filter. No equation or definition in the abstract reduces the claimed subspace direction to a fitted parameter or to the extrapolation step itself; the theoretical grounding is presented as explanatory rather than tautological. Experiments on GPT-2, LLaMA, and Pythia models supply external performance checks that are not forced by the observation alone. No self-citation chain, ansatz smuggling, or renaming of known results is exhibited.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract provides no explicit free parameters, axioms, or invented entities; full text would be required to audit them.

pith-pipeline@v0.9.1-grok · 5737 in / 991 out tokens · 26223 ms · 2026-06-29T19:45:13.216126+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

5 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    PMLR, 2020. Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 12 20...

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  2. [2]

    (Eigengap)There exist β >0 and β♭ >0 such that for all w∈U , |λd(∇2L(w))|< β ♭ and λd−1(∇2L(w))> β+ 4β ♭

  3. [3]

    straight river

    (Slow spinning)There exist ρ∈[0,0.01) and 0< ν min ≤ν such that for all w∈U , νmin <∥∇L(w)∥ 2 2 ≤ν and ∥∇v(w)∥op ≤ρ· β 2ν . 16 Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training To obtain closed-form(N, T)-dependent bounds, we adopt the standard “straight river” simplification. Assumption C.2(Straight river on U).On U...

  4. [4]

    Since the process is stationary and zero-mean,E[(¯u(j))2] = Var(¯u(j))

    By orthogonality of the basis {qj}, this isPd−1 j=1 E[(¯u(j))2]. Since the process is stationary and zero-mean,E[(¯u(j))2] = Var(¯u(j)). Var(¯u(j)) = Var 1 N N−1X m=0 u(j) k0+mT ! = 1 N2 N−1X m=0 N−1X n=0 Cov(u(j) k0+mT , u(j) k0+nT ) = 1 N2 N−1X m=0 N−1X n=0 a|m−n|T j Var(u(j)) = Var(u(j)) N2  X m=n a0·T j + X m̸=n a|m−n|T j   . The double summation ...

  5. [5]

    Since the mountain process is stationary and zero-mean,E[ε s] = 0

    Therefore, E[∥εs −¯ε∥2 2]≤E[∥ε s −c∥ 2 2] for any constant vector c. Since the mountain process is stationary and zero-mean,E[ε s] = 0. Choosingc= 0gives: E[∥˜εs∥2 2]≤E[∥ε s∥2 2]. Finally, by definition,ε s =P S( ¯w(s) −w ⋆), so its squared norm is the squared river deviation: E[∥εs∥2 2] =E[∥P S( ¯w(s) −w ⋆)∥2 2] =ED( ¯w(s))2. Combining these inequalities...