pith. sign in

arxiv: 2604.25965 · v2 · pith:BVZE2YXNnew · submitted 2026-04-28 · 📊 stat.ML · cs.LG

Adversarial Robustness of NTK Neural Networks

Yuxuan Hou This is my paper

Pith reviewed 2026-07-01 09:00 UTC · model grok-4.3

classification 📊 stat.ML cs.LG
keywords adversarial robustnessneural tangent kernelnonparametric regressionSobolev spacesminimax ratesgradient flowearly stopping
0
0 comments X

The pith

NTK neural networks achieve minimax optimal rates for adversarial regression in Sobolev spaces when trained via gradient flow with early stopping.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that neural networks in the neural tangent kernel regime can match the best possible rates for recovering functions from data even when inputs can be adversarially perturbed, provided training uses gradient flow and early stopping. It first derives those optimal rates for Sobolev function spaces and then proves the networks reach them, while showing that the minimum-norm interpolant fails under the same attacks. A sympathetic reader would care because the result separates the robustness properties of properly stopped kernel-like training from the vulnerabilities that arise in the overfitting regime.

Core claim

NTK neural networks, trained via gradient flow with early stopping, can achieve the minimax optimal rates for adversarial regression in Sobolev spaces. However, in the overfitting regime, the minimum norm interpolant is vulnerable to adversarial perturbations.

What carries the argument

Gradient flow training with early stopping inside the neural tangent kernel regime of infinite-width networks.

Load-bearing premise

The neural tangent kernel limit accurately describes the training dynamics of the networks under study.

What would settle it

An explicit computation or simulation in which the adversarial risk of an early-stopped NTK estimator exceeds the derived minimax rate by more than a constant factor would falsify the central claim.

Figures

Figures reproduced from arXiv: 2604.25965 by Yuxuan Hou.

Figure 1
Figure 1. Figure 1: Evolution of Adversarial Risk RA over training time t in the exact NTK regime. From left to right: 1D Synthetic data with Gaussian noise, real-world Diabetes regression dataset on S d−1 , and High-Dim (d = 5) Synthetic data. The universally consistent U-shaped curves highlight the fundamental necessity of early stopping (or equivalent spectral regularization) to prevent the severe degradation of adversaria… view at source ↗
Figure 2
Figure 2. Figure 2: Left: Training dynamics of a wide ReLU network. Right: Function space visualization view at source ↗
Figure 3
Figure 3. Figure 3: Evolution of Adversarial Risk RA over training time t with α-trimming smoothing. 14 view at source ↗
read the original abstract

Deep learning models are widely deployed in safety-critical domains, but remain vulnerable to adversarial attacks. In this paper, we study the adversarial robustness of NTK neural networks in the context of nonparametric regression. We establish minimax optimal rates for adversarial regression in Sobolev spaces and then show that NTK neural networks, trained via gradient flow with early stopping, can achieve this optimal rate. However, in the overfitting regime, we prove that the minimum norm interpolant is vulnerable to adversarial perturbations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 2 minor

Summary. The manuscript establishes minimax optimal rates for adversarial regression over Sobolev balls and shows that NTK neural networks trained by gradient flow with early stopping attain these rates. It further proves that the minimum-norm interpolant fails to be robust in the overfitting regime.

Significance. If the derivations hold, the work supplies a clean nonparametric-statistical account of adversarial robustness for kernel methods in the NTK limit, with an explicit separation between early-stopped gradient flow and interpolation. The result is of interest to both the adversarial-robustness and nonparametric-statistics communities.

minor comments (2)
  1. [Abstract] Abstract: the statement of the Sobolev-ball rates omits the dependence on smoothness index s and dimension d; these parameters should appear explicitly so that the claimed optimality is immediately verifiable.
  2. The transition from the population minimax analysis to the finite-sample NTK gradient-flow analysis would benefit from a short paragraph clarifying the uniform control on the NTK approximation error that is used to transfer the population rates to the empirical setting.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive evaluation of the manuscript, accurate summary of our contributions, and recommendation for minor revision. We are pleased that the work is viewed as providing a clean nonparametric account of adversarial robustness for NTK methods.

Circularity Check

0 steps flagged

No significant circularity; derivation is self-contained

full rationale

The paper first derives minimax rates for adversarial regression over Sobolev balls using standard nonparametric analysis, then separately shows that NTK gradient flow with early stopping attains those rates. No step reduces a claimed prediction or first-principles result to a fitted input, self-citation, or definitional tautology. The additional claim about minimum-norm interpolants in the overfitting regime is consistent with known kernel results and does not rely on the main result. The derivation chain therefore remains independent of its own outputs.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

Ledger is based on the abstract only; full paper likely specifies more technical assumptions on the kernel and data distribution.

axioms (2)
  • domain assumption The regression functions lie in Sobolev spaces
    Defines the function class for which minimax rates are derived.
  • domain assumption The neural network operates in the NTK regime
    Allows the use of kernel methods for analysis.

pith-pipeline@v0.9.1-grok · 5590 in / 1117 out tokens · 36406 ms · 2026-07-01T09:00:41.748920+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 12 canonical work pages · 2 internal anchors

  1. [1]

    doi: 10.1214/22-EJS2011

    ISSN 1935-7524. doi: 10.1214/22-EJS2011. Haim Brezis.Functional Analysis, Sobolev Spaces and Partial Differential Equations. Springer New York,

    work page doi:10.1214/22-ejs2011 1935

  2. [2]

    doi: 10.1007/978-0-387-70914-7

    ISBN 978-0-387-70913-0. doi: 10.1007/978-0-387-70914-7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    work page doi:10.1007/978-0-387-70914-7 1901

  3. [3]

    Trudinger.Elliptic Partial Differential Equations of Second Order

    David Gilbarg and Neil S. Trudinger.Elliptic Partial Differential Equations of Second Order. Classics in Mathematics. Springer-Verlag Berlin Heidelberg, reprint of the 1998 edition edition,

    1998

  4. [4]

    , TITLE =

    ISBN 978-3-540-41160-4. doi: 10.1007/978-3-642-61798-0. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neu- ral networks. InProceedings of the thirteenth international conference on artificial intelligence and statistics (AISTATS), pages 249–256. JMLR Workshop and Conference Proceedings,

    work page doi:10.1007/978-3-642-61798-0

  5. [5]

    On the effectiveness of interval bound propagation for training verifiably robust models,

    Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. On the effectiveness of inter- val bound propagation for training verifiably robust models.arXiv preprint arXiv:1810.12715,

    work page arXiv

  6. [6]

    Moritz Haas, David Holzmüller, Ulrike von Luxburg, and Ingo Steinwart

    doi: 10.1007/978-1-4939-1230-8. Moritz Haas, David Holzmüller, Ulrike von Luxburg, and Ingo Steinwart. Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension. InAdvances in Neural Information Processing Systems, volume 36, pages 54406– 54437,

    work page doi:10.1007/978-1-4939-1230-8

  7. [7]

    Yifan Hao and Tong Zhang

    URLhttps://proceedings.neurips.cc/paper_files/paper/2023/hash/ ab6526e0388279374024467a33605342-Abstract-Conference.html. Yifan Hao and Tong Zhang. The surprising harmfulness of benign overfitting for adversarial robustness,

    2023

  8. [8]

    arXiv:2401.12236

    URLhttps://arxiv.org/abs/2401.12236. arXiv:2401.12236. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 770–778,

    work page arXiv

  9. [9]

    Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences

    MotonobuKanagawa, PhilippHennig, DinoSejdinovic, andBharathKSriperumbudur. Gaussian processes and kernel methods: A review on connections and equivalences.arXiv preprint arXiv:1807.02582,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    Generalization ability of wide neural networks onR.arXiv preprint arXiv:2302.05933, 2023

    URLhttps://arxiv.org/abs/2302.05933. SubmittedtotheAnnalsofStatistics. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition.Proceedings of the IEEE, 86(11):2278–2324,

    work page arXiv

  11. [11]

    Adversarial robustness of nonparametric regression.arXiv preprint arXiv:2505.17356,

    Parsa Moradi, Hanzaleh Akabrinodehi, and Mohammad Ali Maddah-Ali. Adversarial robustness of nonparametric regression.arXiv preprint arXiv:2505.17356,

    work page arXiv

  12. [12]

    Minimax rates of convergence for nonparametric regression under adversarial attacks.arXiv preprint arXiv:2410.09402,

    Jingfu Peng and Yuhong Yang. Minimax rates of convergence for nonparametric regression under adversarial attacks.arXiv preprint arXiv:2410.09402,

    work page arXiv

  13. [14]

    Jingfu Peng and Yuhong Yang

    arXiv:2506.01267v1. Jingfu Peng and Yuhong Yang. On damage of interpolation to adversarial robustness in regres- sion.arXiv preprint arXiv:2601.16070,

    work page arXiv

  14. [16]

    Understanding deep learning requires rethinking generalization

    URL https://arxiv.org/abs/1611.03530. Haobo Zhang, Yicheng Li, and Qian Lin. On the optimality of misspecified spectral algorithms. Journal of Machine Learning Research, 25:1–50,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [17]

    Proof.We will prove the convexity and closedness of the Sobolev ballHs(L) ={f∈H s([0,1] d) : ∥f∥ H s ≤L}inL 2([0,1] d)separately

    A Proof of Section 3 A.1 Upper Bound Lemma A.1.The Sobolev ballH s(L)is a closed and convex subset ofL2([0,1] d). Proof.We will prove the convexity and closedness of the Sobolev ballHs(L) ={f∈H s([0,1] d) : ∥f∥ H s ≤L}inL 2([0,1] d)separately. Step 1: ConvexityLetf, g∈ H s(L)andt∈[0,1]. We aim to show that the convex combinationh=tf+ (1−t)galso belongs to...

    2011

  16. [18]

    Corollary A.3(Adapted from Zhang et al

    directly implies the following result by simply taking an integral. Corollary A.3(Adapted from Zhang et al. [2024]). sup f∈H s(L) EDn∥ ˜f−f∥ 2 L2 ≲n − 2s 2s+d . Proof.LetX=∥ ˜f−f∥ 2 L2. Theorem 1 from Zhang et al

    2024

  17. [19]

    sup x′∈A(X) |g(x′)−g(X)| 2 # ≤r 2(1∧s)∥g∥2 H s. Combining Theorem A.6 and Corollary A.2, we obtain the desired bound: RA( ˆf , f)≲E

    =ϵ 2 n log2(6) + 2 log(6) + 2 . Sincelog 2(6) + 2 log(6) + 2is an absolute constant, we conclude that: E∥ ˜f−f∥ 2 L2 ≤C s log2(6) + 2 log(6) + 2 n− 2s 2s+d ≲n − 2s 2s+d . This completes the proof. A.1.1 Proof ofd= 1,s > d/2 Using the inequality(a+b) 2 ≤2a 2 + 2b2, we decompose the adversarial risk: RA( ˆf , f) =E X,Dn[ sup x′∈A(X) | ˆf(x ′)− ˆf(X) + ˆf(X)...

    2001

  18. [20]

    LetU=B(x 0, r/2)and define the local essential supremumSn = supx′∈U | ˆfn(x′)|

    Since the adversarial risk is lower bounded by the standardL2 risk, takingr→0impliesE∥ ˆfn −f ∗∥2 L2 ≤ Cn− 2s 2s+d. LetU=B(x 0, r/2)and define the local essential supremumSn = supx′∈U | ˆfn(x′)|. Sincef ∗ is essentially unbounded onU, for any arbitrarily large constantM >0, the truncation error lower boundC M = inf ∥g∥L∞(U) ≤M ∥g−f ∗∥2 L2(U) is strictly p...

    2024

  19. [21]

    Thus, the NTK kernel and the exponential kernelk(x, y) =e−|x−y| are equivalent in a bounded smooth boundary domain, also in its subdomain

    and the NTK kernel in this paper are different up to adding 1, and noticing that 1 lies in the Sobolev space, for the NTK kernel in our setting, we also have that the RKHS of NTK in a bounded domain with smooth boundary is a Sobolev class. Thus, the NTK kernel and the exponential kernelk(x, y) =e−|x−y| are equivalent in a bounded smooth boundary domain, a...

    2018