REVIEW 2 major objections 1 minor 299 references
USAD detects adversarial examples more reliably by measuring their excess feature variance and instability under perturbations.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.3
2026-06-29 04:48 UTC pith:GCFM2QDW
load-bearing objection USAD defines two new discrepancy stats (VD and PCD) to target uncertainty patterns missed by MMD in SAD, but the abstract gives no experimental backing for the performance claims. the 2 major comments →
USAD: Uncertainty-aware Statistical Adversarial Detection
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
USAD augments the statistical adversarial detection framework with Variance Discrepancy, which measures the difference in feature variance between queries and clean examples, and Perturbation-based Covariance Discrepancy, which measures the change in feature covariance after adding Gaussian noise; their combination yields a stronger two-sample test statistic than MMD for identifying batches that contain adversarial examples.
What carries the argument
Variance Discrepancy (VD) and Perturbation-based Covariance Discrepancy (PCD) aggregated as uncertainty-aware test statistics inside the two-sample testing procedure for SAD.
Load-bearing premise
Adversarial examples reliably produce larger feature spread and greater covariance instability under perturbations than clean examples, and these differences are not already captured by MMD.
What would settle it
A controlled experiment in which adversarial examples generated against the same model show equal or smaller variance and equal or smaller covariance change under Gaussian perturbation than clean examples, resulting in no detection gain for USAD over MMD.
If this is right
- VD isolates global uncertainty by comparing raw feature variances across the two sets.
- PCD isolates local uncertainty by comparing covariance matrices computed on original and perturbed inputs.
- The combined statistic improves detection accuracy over MMD baselines on multiple attack types while retaining type-I error control.
- The method demonstrates that SAD performance rises when the test statistic is chosen to match known properties of the shift being detected.
Where Pith is reading between the lines
- The same uncertainty statistics might be applied to detect other non-adversarial distribution shifts that also increase feature spread or instability.
- Alternative perturbation distributions or higher-order moments could be substituted for the Gaussian noise used in PCD.
- The two-sample testing view suggests that any domain-specific signature of a shift can be turned into a custom discrepancy measure.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes Uncertainty-aware Statistical Adversarial Detection (USAD) as an extension of statistical adversarial detection (SAD). It introduces two new discrepancy statistics—Variance Discrepancy (VD) to capture global feature-spread differences and Perturbation-based Covariance Discrepancy (PCD) to capture local covariance instability under Gaussian perturbations—then aggregates them to detect distributional drift between clean examples and a query batch that may contain adversarial examples. The central claim is that this aggregation yields superior detection performance over MMD-based baselines on various attacks while preserving false-alarm control.
Significance. If the empirical results hold and the new statistics integrate cleanly into the two-sample test without new failure modes, the work could strengthen SAD by explicitly targeting AE-specific uncertainty behaviors that standard MMD may miss. The public code link supports reproducibility. However, the provided abstract contains no quantitative results, datasets, error bars, or ablation studies, so the practical significance cannot be assessed from the given text.
major comments (2)
- [Abstract] Abstract: the central empirical claim that 'USAD achieves superior detection performances over baseline methods' is stated without any experimental details, error bars, dataset descriptions, ablation results, or statistical significance tests. This is load-bearing for the paper's main contribution.
- [Abstract] Abstract: the claim that VD and PCD capture uncertainty patterns 'crucial for detection' and not already addressed by MMD rests on an unverified assumption; no derivation, proof, or preliminary comparison is supplied to show that the new statistics are independent of or additive to MMD.
minor comments (1)
- [Abstract] The code repository link uses an anonymous domain; a permanent, non-anonymous link should be provided for review.
Simulated Author's Rebuttal
We thank the referee for their review and constructive comments on the abstract. We address each major comment point by point below, indicating planned revisions where appropriate.
read point-by-point responses
-
Referee: [Abstract] Abstract: the central empirical claim that 'USAD achieves superior detection performances over baseline methods' is stated without any experimental details, error bars, dataset descriptions, ablation results, or statistical significance tests. This is load-bearing for the paper's main contribution.
Authors: We agree that the abstract presents the empirical claim at a summary level without quantitative details. This follows standard abstract conventions given length limits, while the full manuscript reports all requested elements (error bars, datasets, ablations, and significance tests) in the Experiments section. To address the concern directly, we will revise the abstract to incorporate a concise statement of key results and datasets used. revision: yes
-
Referee: [Abstract] Abstract: the claim that VD and PCD capture uncertainty patterns 'crucial for detection' and not already addressed by MMD rests on an unverified assumption; no derivation, proof, or preliminary comparison is supplied to show that the new statistics are independent of or additive to MMD.
Authors: The abstract summarizes the motivation that MMD's properties limit capture of AE-specific global spread and local perturbation instability; the full manuscript provides the supporting conceptual analysis in Section 3 and demonstrates additivity via ablations in Section 5. No formal independence proof is given, as the contribution is empirical and statistical rather than theoretical. We will add a short clarifying sentence referencing the empirical additivity in the revised abstract. revision: partial
Circularity Check
No significant circularity identified
full rationale
The provided text consists only of the abstract, which introduces two new statistics (VD and PCD) as explicit measures of global and local uncertainty patterns without any equations, derivations, parameter fitting, or self-citations. No step reduces a claimed result to its inputs by construction, renames a known result, or relies on load-bearing self-citation. The central claim is an empirical performance comparison, which remains independent of the inputs in the given material. This matches the reader's assessment of score 2.0 with no equations shown that could create circularity.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Two-sample test framework with controlled false-alarm rate applies to mixed clean/adversarial query batches
read the original abstract
Statistical adversarial detection (SAD) treats detection as a two-sample test. Given a reference set of clean examples (CEs) and a batch of queries, potentially containing an unknown mixture of CEs and adversarial examples (AEs), SAD decides whether the query distribution drifts away from the CE distribution while controlling the false-alarm rate. Existing SAD-based methods mainly use maximum mean discrepancy (MMD) to measure the distributional discrepancy. However, MMD's distributional properties limit its ability to capture characteristic uncertainty patterns of AEs that are crucial for detection: AEs typically exhibit abnormal feature spread (i.e., global uncertainty) and instability under perturbations (i.e., local uncertainty). To close the gap, we propose Uncertainty-aware Statistical Adversarial Detection (USAD), which explicitly captures these uncertainty patterns with two new statistics: (1) Variance Discrepancy (VD), which measures the difference in feature spread between AEs and CEs to capture global uncertainty differences. (2) Perturbation-based Covariance Discrepancy (PCD), which compares feature covariance under Gaussian perturbations to capture local uncertainty differences. By aggregating VD and PCD, USAD achieves superior detection performances over baseline methods against various adversarial attacks, highlighting the importance of considering characteristic behaviors of AEs for effective SAD. Our code is available at: https://anonymous.4open.science/r/USAD.
Figures
Reference graph
Works this paper leans on
-
[1]
Square attack: A query-efficient black-box adversarial attack via random search
Andriushchenko, M., Croce, F., Flammarion, N., and Hein, M. Square attack: A query-efficient black-box adversarial attack via random search. In ECCV, 2020
2020
-
[2]
MMD-Fuse : Learning and combining kernels for two-sample testing without data splitting
Biggs, F., Schrab, A., and Gretton, A. MMD-Fuse : Learning and combining kernels for two-sample testing without data splitting. In NeurIPS, 2023
2023
-
[3]
Evading adversarial example detection defenses with orthogonal projected gradient descent
Bryniarski, O., Hingun, N., Pachuca, P., Wang, V., and Carlini, N. Evading adversarial example detection defenses with orthogonal projected gradient descent. In ICLR, 2022
2022
-
[4]
A., Liu, M., and Li, B
Cao, Y., Wang, N., Xiao, C., Yang, D., Fang, J., Yang, R., Chen, Q. A., Liu, M., and Li, B. Invisible for both camera and lidar: Security of multi-sensor fusion based perception in autonomous driving under physical-world attacks. In IEEE Symposium on Security and Privacy , pp.\ 176--194, 2021
2021
-
[5]
and Wagner, D
Carlini, N. and Wagner, D. Adversarial examples are not easily detected: Bypassing ten detection methods. In AISec, 2017 a
2017
-
[6]
and Wagner, D
Carlini, N. and Wagner, D. A. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy , pp.\ 39--57. IEEE, 2017 b
2017
-
[7]
and Bhattacharya, B
Chatterjee, A. and Bhattacharya, B. B. Boosting the power of kernel two-sample tests. Biometrika, 112 0 (1): 0 asae048, 2025
2025
-
[8]
and Cloninger, A
Cheng, X.-Y. and Cloninger, A. Classification logit two-sample testing by neural networks. arXiv, 2019
2019
-
[9]
Certified adversarial robustness via randomized smoothing
Cohen, J., Rosenfeld, E., and Kolter, Z. Certified adversarial robustness via randomized smoothing. In ICML, 2019
2019
-
[10]
and Hein, M
Croce, F. and Hein, M. Minimally distorted adversarial examples with a fast adaptive boundary attack. In ICML, 2020 a
2020
-
[11]
and Hein, M
Croce, F. and Hein, M. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, 2020 b
2020
-
[12]
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR, 2009
2009
-
[13]
LiBRe : A practical B ayesian approach to adversarial detection
Deng, Z., Yang, X., Xu, S., Su, H., and Zhu, J. LiBRe : A practical B ayesian approach to adversarial detection. In CVPR, 2021
2021
-
[14]
An image is worth 16x16 words: Transformers for image recognition at scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021
2021
-
[15]
Robustness of classifiers: from adversarial to random noise
Fawzi, A., Moosavi-Dezfooli, S.-M., and Frossard, P. Robustness of classifiers: from adversarial to random noise. In NeurIPS, 2016
2016
-
[16]
R., Shintre, S., and Gardner, A
Feinman, R., Curtin, R. R., Shintre, S., and Gardner, A. B. Detecting adversarial samples from artifacts. arXiv, 2017
2017
-
[17]
Maximum mean discrepancy test is aware of adversarial attacks
Gao, R., Liu, F., Zhang, J., Han, B., Liu, T., Niu, G., and Sugiyama, M. Maximum mean discrepancy test is aware of adversarial attacks. In ICML, 2021
2021
-
[18]
J., Shlens, J., and Szegedy, C
Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. In ICLR, 2015
2015
-
[19]
M., Rasch, M
Gretton, A., Borgwardt, K. M., Rasch, M. J., Sch \"o lkopf, B., and Smola, A. A kernel two-sample test. Journal of Machine Learning Research, 13 0 (1): 0 723--773, 2012
2012
-
[20]
On the (statistical) detection of adversarial examples
Grosse, K., Manoharan, P., Papernot, N., Backes, M., and McDaniel, P. On the (statistical) detection of adversarial examples. arXiv, 2017
2017
-
[21]
Trustworthy Machine Learning: From Data to Models
Han, B., Yao, J., Liu, T., Li, B., Koyejo, S., and Liu, F. Trustworthy Machine Learning: From Data to Models. Now Foundations and Trends, 2025
2025
-
[22]
Kernel change-point analysis
Harchaoui, Z., Moulines, E., and Bach, F. Kernel change-point analysis. In NeurIPS, 2008
2008
-
[23]
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, 2016
2016
-
[24]
and Goeman, J
Hemerik, J. and Goeman, J. Exact testing with random permutations. Test, 27 0 (4): 0 811--825, 2018
2018
-
[25]
Probability inequalities for sums of bounded random variables
Hoeffding, W. Probability inequalities for sums of bounded random variables. In The Collected Works of Wassily Hoeffding, pp.\ 409--426, New York, NY, 1994. Springer
1994
-
[26]
Model-agnostic adversarial detection by random perturbations
Huang, B., Wang, Y., and Wang, W. Model-agnostic adversarial detection by random perturbations. In IJCAI, 2019
2019
-
[27]
H., and Courville, A
Huang, C., Lim, J. H., and Courville, A. C. A variational perspective on diffusion-based generative models and score matching. In NeurIPS, 2021
2021
-
[28]
Adversarial examples are not bugs, they are features
Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., and Madry, A. Adversarial examples are not bugs, they are features. In NeurIPS, 2019
2019
-
[29]
Too good to be safe: Tricking lane detection in autonomous driving with crafted perturbations
Jing, P., Tang, Q., Du, Y., Xue, L., Luo, X., Wang, T., Nie, S., and Wu, S. Too good to be safe: Tricking lane detection in autonomous driving with crafted perturbations. In USENIX Security Symposium , pp.\ 3237--3254, 2021
2021
-
[30]
Diffattack: Evasion attacks against diffusion-based adversarial purification
Kang, M., Song, D., and Li, B. Diffattack: Evasion attacks against diffusion-based adversarial purification. In NeurIPS, 2023
2023
-
[31]
B., Aydore, S., Rauschmayr, N., and Kenthapadi, K
Kaya, Y., Zafar, M. B., Aydore, S., Rauschmayr, N., and Kenthapadi, K. Generating distributional adversarial examples to evade statistical detectors. In ICML, 2022
2022
-
[32]
Minimax optimality of permutation tests
Kim, I., Balakrishnan, S., and Wasserman, L. Minimax optimality of permutation tests. Annals of Statistics, 50 0 (1): 0 225--251, 2022
2022
-
[33]
CIFAR-10 (canadian institute for advanced research)
Krizhevsky, A., Nair, V., and Hinton, G. CIFAR-10 (canadian institute for advanced research). 2009. URL http://www.cs.toronto.edu/ kriz/cifar.html
2009
-
[34]
J., and Bengio, S
Kurakin, A., Goodfellow, I. J., and Bengio, S. Adversarial examples in the physical world. In ICLR, Workshop Track Proceedings, 2017
2017
-
[35]
Certified adversarial robustness with additive noise
Li, B., Chen, C., Wang, W., and Carin, L. Certified adversarial robustness with additive noise. In NeurIPS, 2019
2019
-
[36]
Liu, F., Xu, W., Lu, J., Zhang, G., Gretton, A., and Sutherland, D. J. Learning deep kernels for non-parametric two-sample tests. In ICML, 2020
2020
-
[37]
M., Wijewickrema, S
Ma, X., Li, B., Wang, Y., Erfani, S. M., Wijewickrema, S. N. R., Schoenebeck, G., Song, D., Houle, M. E., and Bailey, J. Characterizing adversarial subspaces using local intrinsic dimensionality. In ICLR, 2018
2018
-
[38]
Towards deep learning models resistant to adversarial attacks
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018
2018
-
[39]
DeepFool : A simple and accurate method to fool deep neural networks
Moosavi-Dezfooli, S.-M., Fawzi, A., and Frossard, P. DeepFool : A simple and accurate method to fool deep neural networks. In CVPR, 2016
2016
-
[40]
Robustness of classifiers to universal perturbations: A geometric perspective
Moosavi-Dezfooli, S.-M., Fawzi, A., Fawzi, O., Frossard, P., and Soatto, S. Robustness of classifiers to universal perturbations: A geometric perspective. In ICLR, 2018
2018
-
[41]
Kernel mean embedding of distributions: A review and beyond
Muandet, K., Fukumizu, K., Sriperumbudur, B., and Sch \"o lkopf, B. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends in Machine Learning , 10 0 (1-2): 0 1--141, 2017
2017
-
[42]
and Pillai, S
Papoulis, A. and Pillai, S. U. Probability, Random Variables and Stochastic Processes. McGraw Hill, 2001
2001
-
[43]
The odds are odd: A statistical test for detecting adversarial examples
Roth, K., Kilcher, Y., and Hofmann, T. The odds are odd: A statistical test for detecting adversarial examples. In ICML, 2019
2019
-
[44]
Efficient aggregated kernel tests using incomplete U -statistics
Schrab, A., Kim, I., Guedj, B., and Gretton, A. Efficient aggregated kernel tests using incomplete U -statistics. In NeurIPS, 2022
2022
-
[45]
MMD aggregated two-sample test
Schrab, A., Kim, I., Albert, M., Laurent, B., Guedj, B., and Gretton, A. MMD aggregated two-sample test. Journal of Machine Learning Research, 24: 0 194:1--194:81, 2023
2023
-
[46]
and Ermon, S
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. In NeurIPS, 2019
2019
-
[47]
P., Kumar, A., Ermon, S., and Poole, B
Song, Y., Sohl - Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In ICLR, 2021
2021
-
[48]
Sample-specific noise injection for diffusion-based adversarial purification
Sun, Y., Zhang, J., Ye, Z., Xiao, C., and Liu, F. Sample-specific noise injection for diffusion-based adversarial purification. In ICML, 2025
2025
-
[49]
J., and Fergus, R
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. J., and Fergus, R. Intriguing properties of neural networks. In ICLR, 2014
2014
-
[50]
Detecting adversarial examples is (nearly) as hard as classifying them
Tram \` e r, F. Detecting adversarial examples is (nearly) as hard as classifying them. In ICML, 2022
2022
-
[51]
Sparse adversarial perturbations for videos
Wei, X., Zhu, J., Yuan, S., and Su, H. Sparse adversarial perturbations for videos. In AAAI, 2019
2019
-
[52]
Diffusion-based adversarial sample generation for improved stealthiness and controllability
Xue, H., Araujo, A., Hu, B., and Chen, Y. Diffusion-based adversarial sample generation for improved stealthiness and controllability. In NeurIPS, 2023
2023
-
[53]
On the certified robustness for ensemble models and beyond
Yang, Z., Li, L., Xu, X., Kailkhura, B., Xie, T., and Li, B. On the certified robustness for ensemble models and beyond. In ICLR, 2022
2022
-
[54]
P., Ghaoui, L
Zhang, H., Yu, Y., Jiao, J., Xing, E. P., Ghaoui, L. E., and Jordan, M. I. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019
2019
-
[55]
Improving accuracy-robustness trade-off via pixel reweighted adversarial training
Zhang, J., Liu, F., Zhou, D., Zhang, J., and Liu, T. Improving accuracy-robustness trade-off via pixel reweighted adversarial training. In ICML, 2024
2024
-
[56]
Zhang, J., Rubinstein, B. I. P., Zhang, J., and Liu, F. One stone, two birds: Enhancing adversarial defense through the lens of distributional discrepancy. In ICML, 2025
2025
-
[57]
Detecting adversarial data by probing multiple perturbations using expected perturbation score
Zhang, S., Liu, F., Yang, J., Yang, Y., Li, C., Han, B., and Tan, M. Detecting adversarial data by probing multiple perturbations using expected perturbation score. In ICML, 2023
2023
-
[58]
and Liu, C
Zhong, X. and Liu, C. Sparse-pgd: A unified framework for sparse adversarial perturbations generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025
2025
-
[59]
On the exploration of local significant differences for two-sample test
Zhou, Z., Ni, J., Yao, J.-H., and Gao, W. On the exploration of local significant differences for two-sample test. In NeurIPS, 2023
2023
-
[60]
J., and Liu, F
Zhou, Z., Tian, X., Peng, L., Lei, C., Schrab, A., Sutherland, D. J., and Liu, F. DUAL : Learning diverse kernels for aggregated two-sample and independence testing. In NeurIPS, 2025
2025
-
[61]
ICML , year=
The odds are odd: A statistical test for detecting adversarial examples , author=. ICML , year=
-
[62]
, author=
Model-Agnostic Adversarial Detection by Random Perturbations. , author=. IJCAI , year=
-
[63]
NeurIPS , year=
Robustness of classifiers: from adversarial to random noise , author=. NeurIPS , year=
-
[64]
ICML , year=
Generating Distributional Adversarial Examples to Evade Statistical Detectors , author=. ICML , year=
-
[65]
Adversarial Examples Are Not Bugs, They Are Features , author =
-
[66]
Robustness of Classifiers to Universal Perturbations:
Moosavi-Dezfooli, Seyed-Mohsen and Fawzi, Alhussein and Fawzi, Omar and Frossard, Pascal and Soatto, Stefano , year = 2018, booktitle =. Robustness of Classifiers to Universal Perturbations:
2018
-
[67]
Moosavi-Dezfooli, Seyed-Mohsen and Fawzi, Alhussein and Frossard, Pascal , year = 2016, booktitle =
2016
-
[68]
Certified Adversarial Robustness via Randomized Smoothing , author =
-
[69]
Certified Adversarial Robustness with Additive Noise , author =
-
[70]
On the Certified Robustness for Ensemble Models and Beyond , author =
-
[71]
Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods , author =
-
[72]
Deng, Zhijie and Yang, Xiao and Xu, Shizhen and Su, Hang and Zhu, Jun , year = 2021, booktitle =. LiBRe:
2021
-
[73]
A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks , author =
-
[74]
Detecting adversarial samples from artifacts , author =
-
[75]
Two-sample Testing Using Deep Learning , author =
-
[76]
Kernel Change-point Analysis , author =
-
[77]
Testing for Homogeneity with Kernel Fisher Discriminant Analysis , author =
-
[78]
Annals of Statistics , volume = 52, number = 3, pages =
Spectral regularized kernel two-sample tests , author =. Annals of Statistics , volume = 52, number = 3, pages =
-
[79]
Is Out-of-Distribution Detection Learnable? , author =
-
[80]
Dataset shift in machine learning , author =
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.