pith. sign in

arxiv: 2605.11554 · v2 · pith:PS6VCTOFnew · submitted 2026-05-12 · 💻 cs.LG

A Controlled Counterexample to Strong Proxy-Based Explanations of OOD Performance: in a Fixed Pretraining-and-Probing Setup

Hongmin Li This is my paper

Pith reviewed 2026-07-01 07:52 UTC · model grok-4.3

classification 💻 cs.LG
keywords out-of-distribution generalizationpretrainingstructure proxiescounterexampletransfer learningsequence models
0
0 comments X

The pith

A structure proxy ranking pretraining datasets can reverse the OOD accuracy ranking they are meant to explain.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether task-agnostic proxies for learned structure reliably predict which pretraining corpus will produce better out-of-distribution performance on a downstream probe. In a fixed pretraining-and-probing setup, it constructs a case where a formal structure quantity, its measurable proxy, and the structure actually needed for the target task can be made to separate. The same separation is then realized in a synthetic sequence-model experiment where, under the main evaluation, the proxy ranking of two datasets reverses their OOD accuracy ranking in two of three random seeds. The result shows that strong proxy-based explanations of transfer have a boundary condition even when everything else is held constant.

Core claim

In a controlled pretraining-and-probing setup, a proxy for total learned structure need not preserve the ranking of datasets by out-of-distribution probe accuracy, because the proxy can be made to track structure that is irrelevant to the downstream task while missing the structure that matters.

What carries the argument

The controlled construction in which a formal structure quantity, its operational proxy, and the task-relevant structure for a target family are separated from one another.

If this is right

  • Explanations that rely on a single aggregate structure proxy cannot be assumed to track downstream utility without additional checks that the proxy aligns with task-relevant structure.
  • Controlled counterexamples of this form can be used to delineate when proxy-based accounts of transfer remain reliable.
  • Auxiliary diagnostics that measure alignment between proxy and task structure become necessary before interpreting transfer differences.
  • The boundary identified here applies even when pretraining method, probe architecture, and evaluation protocol are held fixed.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Researchers comparing corpora may need to measure multiple proxies or directly estimate task alignment rather than relying on any single total-structure metric.
  • The same separation mechanism could be tested in larger-scale language-model pretraining to check whether the boundary scales beyond the synthetic regime.

Load-bearing premise

The synthetic experiment genuinely separates the proxy from the task-relevant structure rather than introducing an artifact of the data generation process.

What would settle it

Re-running the synthetic experiment with the same architecture and data-generation rules but different random seeds and finding that the proxy ranking and OOD accuracy ranking agree in every seed under the primary evaluation metric.

Figures

Figures reproduced from arXiv: 2605.11554 by Hongmin Li.

Figure 1
Figure 1. Figure 1: Conceptual overview of the controlled counterexample. Dataset A contains more total [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Per-seed OOD probe-accuracy gaps for the primary evaluation. The left panel shows the [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Config-level comparison between the primary evaluation and the background-only ablation. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
read the original abstract

Task-agnostic structure proxies are often used to interpret why one pretraining corpus transfers better than another, but such explanations require the proxy to track the structure that matters for the downstream task. We test this requirement in a fixed pretraining-and-probing setup motivated by computationally bounded notions of learned structure, including epiplexity. The core question is whether a proxy ranking of two pretraining datasets must agree with their ranking by OOD probe accuracy. We show that it need not. First, we give a controlled construction in which a formal structure quantity, its operational proxy, and the task-relevant structure for a target family separate. We then instantiate the same mechanism in a synthetic sequence-model experiment: under the primary all-sample evaluation, the OOD accuracy ranking reverses the proxy ranking in two of three seeds, with auxiliary diagnostics and ablations supporting the same interpretation. The counterexample does not reject structure-based explanations in general; it identifies a boundary on strong proxy-based explanations. A proxy for total learned structure can fail to track the task-relevant structure that drives OOD performance, even in a controlled setting.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 2 minor

Summary. The manuscript claims that task-agnostic structure proxies (e.g., epiplexity) need not track the task-relevant structure that determines OOD probe accuracy in a fixed pretraining-and-probing setup. It supports this via a controlled construction separating a formal structure quantity, its operational proxy, and the task-relevant structure for a target family, then instantiates the mechanism in a synthetic sequence-model experiment where, under the primary all-sample evaluation, the OOD accuracy ranking reverses the proxy ranking in two of three seeds, with auxiliary diagnostics and ablations supporting the interpretation.

Significance. If the construction and experiment hold, the result is significant because it supplies a concrete boundary condition on strong proxy-based explanations of transfer, a common practice in interpretability work. The existence result is strengthened by the use of a parameter-free separation in the construction and by the synthetic instantiation with seed-level reporting and ablations, which together make the counterexample falsifiable and reproducible within the stated scope.

minor comments (2)
  1. [Abstract] The abstract states that the reversal occurs 'in two of three seeds' but does not report the magnitude of the accuracy differences or the proxy values; adding these numbers (even as a parenthetical) would improve immediate readability.
  2. The description of the controlled construction would benefit from an explicit enumeration of the three separated quantities (formal structure, proxy, task-relevant structure) in a single sentence or short list to reduce the chance of misreading the separation.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive summary, significance assessment, and recommendation of minor revision. No major comments were listed in the report, so we have no specific points requiring point-by-point response or changes at this stage. We remain available to address any minor suggestions or clarifications that may arise.

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper presents an existence result via an explicit controlled construction separating a formal structure quantity, its operational proxy, and task-relevant structure, followed by a synthetic instantiation. No derivation chain reduces a claimed prediction to fitted inputs by construction, no load-bearing self-citation justifies the core separation, and no ansatz or uniqueness theorem is smuggled in. The counterexample is internally constructed rather than derived from parameters or prior self-referential results, making the central claim self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The claim rests on the validity of the controlled construction that separates the structure quantity, proxy, and task-relevant structure, and the instantiation in the synthetic experiment.

axioms (2)
  • domain assumption Task-agnostic structure proxies are often used to interpret why one pretraining corpus transfers better than another.
    Opening of abstract.
  • domain assumption The fixed pretraining-and-probing setup is motivated by computationally bounded notions of learned structure, including epiplexity.
    Stated as motivation for the setup.

pith-pipeline@v0.9.1-grok · 5726 in / 1371 out tokens · 47979 ms · 2026-07-01T07:52:17.432879+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

8 extracted references · 2 canonical work pages · 1 internal anchor

  1. [1]

    Understanding intermediate layers using linear classifier probes

    Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  2. [2]

    From entropy to epiplexity: Rethinking infor- mation for computationally bounded intelligence,

    Marc Finzi, Yiding Jiang, J. Zico Kolter, Shikai Qiu, Pavel Izmailov, and Andrew Gordon Wilson. From entropy to epiplexity: Rethinking information for computationally bounded intelligence. arXiv preprint arXiv:2601.03220, 2026

    work page arXiv 2026

  3. [3]

    Data shapley: Equitable valuation of data for machine learning

    Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In Proceedings of the 36th International Conference on Machine Learning, 2019

    2019

  4. [4]

    Gr \"u nwald

    Peter D. Gr \"u nwald. The Minimum Description Length Principle. MIT Press, 2007

    2007

  5. [5]

    Designing and interpreting probes with control tasks

    John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019

    2019

  6. [6]

    Modeling by shortest data description

    Jorma Rissanen. Modeling by shortest data description. Automatica, 14 0 (5): 0 465--471, 1978

    1978

  7. [7]

    Smith, and Yejin Choi

    Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020

    2020

  8. [8]

    Information-theoretic probing with minimum description length

    Elena Voita and Ivan Titov. Information-theoretic probing with minimum description length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020

    2020