pith. sign in

arxiv: 1412.6572 · v3 · pith:I6F6HXECnew · submitted 2014-12-20 · 📊 stat.ML · cs.LG

Explaining and Harnessing Adversarial Examples

Ian J. Goodfellow , Jonathon Shlens , Christian Szegedy This is my paper

Pith reviewed 2026-05-11 04:54 UTC · model grok-4.3

classification 📊 stat.ML cs.LG
keywords adversarial examplesneural networksadversarial traininglinear behaviorgradient sign methodMNIST
0
0 comments X

The pith

Neural networks are vulnerable to adversarial examples mainly because they behave linearly in their inputs.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the main reason neural networks can be fooled by tiny, carefully chosen changes to inputs is their linear response to those inputs rather than any deep nonlinearity or overfitting. This linear view accounts for why the same perturbed examples often fool many different networks and training regimes. It also supplies a fast way to create such examples by following the sign of the input gradient. When those examples are fed back into training, the network's error rate on clean test data drops.

Core claim

The primary cause of neural networks' vulnerability to adversarial perturbation is their linear nature. This linearity explains why adversarial examples generalize across architectures and training sets, and it directly yields a simple, fast method of generating adversarial examples via a first-order approximation that can be used for adversarial training to lower test-set error.

What carries the argument

A first-order linear approximation of the network's output with respect to the input, used to select the direction of perturbation that most increases the loss.

If this is right

  • Adversarial examples generated this way transfer across different network architectures and training sets.
  • Including the generated examples in training reduces test error on the original clean dataset.
  • The same linear approximation explains why the perturbations remain effective even when the network is retrained on different data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If linearity is the root cause, then techniques that enforce stronger local linearity constraints could reduce vulnerability without changing the overall architecture.
  • The approach may extend to other models that exhibit locally linear decision boundaries, such as certain kernel methods or decision trees with linear splits.
  • Defensive training using these examples could be combined with architectural changes that increase curvature to test whether the two strategies are additive.

Load-bearing premise

The network's output changes sufficiently linearly with small input changes that a first-order approximation accurately predicts the effect of a perturbation.

What would settle it

A neural network whose output is demonstrably highly nonlinear for small perturbations yet still produces adversarial examples at rates matching current models, or a linear model that resists them.

read the original abstract

Several machine learning models, including neural networks, consistently misclassify adversarial examples---inputs formed by applying small but intentionally worst-case perturbations to examples from the dataset, such that the perturbed input results in the model outputting an incorrect answer with high confidence. Early attempts at explaining this phenomenon focused on nonlinearity and overfitting. We argue instead that the primary cause of neural networks' vulnerability to adversarial perturbation is their linear nature. This explanation is supported by new quantitative results while giving the first explanation of the most intriguing fact about them: their generalization across architectures and training sets. Moreover, this view yields a simple and fast method of generating adversarial examples. Using this approach to provide examples for adversarial training, we reduce the test set error of a maxout network on the MNIST dataset.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 3 minor

Summary. The paper claims that the vulnerability of neural networks to adversarial examples is primarily due to their linear nature in the input space rather than nonlinearity or overfitting. It supports this via a first-order Taylor approximation motivating the fast gradient sign method for efficient adversarial example generation, shows that such examples transfer across architectures and training sets, and demonstrates that adversarial training reduces test-set error for maxout networks on MNIST.

Significance. If the linearity hypothesis holds in the small-perturbation regime, the work supplies a parsimonious account of cross-model generalization of adversarial examples and yields a computationally cheap attack method plus a practical robustness technique. The MNIST quantitative results are consistent with the claims and the approach has proven influential for subsequent robustness research.

minor comments (3)
  1. [Introduction] Introduction: the reference to 'early attempts' focused on nonlinearity and overfitting would be strengthened by naming the specific prior works being critiqued.
  2. [Fast gradient sign method] Derivation of the fast gradient sign method: a short remark on the range of perturbation magnitudes for which the first-order approximation remains accurate would improve clarity without altering the central argument.
  3. [Experiments] Experiments section: figure captions should explicitly list the value of epsilon used in each panel to facilitate exact reproduction of the reported error rates.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive assessment of the manuscript, the recognition of its significance in providing a parsimonious explanation for the cross-model generalization of adversarial examples, and the recommendation to accept.

Circularity Check

0 steps flagged

No significant circularity in the derivation chain

full rationale

The paper's core argument derives the fast gradient sign method from the first-order Taylor approximation of the network output (f(x + ε) ≈ f(x) + ε · ∇_x f(x)), which is an explicit linearization assumption stated upfront rather than fitted or self-defined. This yields the sign(∇_x J) perturbation without reducing to any input parameter by construction. The claim that linearity is the primary cause is then supported by independent experimental outcomes on MNIST maxout networks (attack success, cross-architecture transfer, and adversarial training gains), none of which loop back to redefine the approximation or rely on self-citations for uniqueness. No enumerated circularity pattern applies; the derivation remains self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim rests on the assumption that neural networks behave linearly for small perturbations; epsilon is a free parameter controlling perturbation magnitude.

free parameters (1)
  • epsilon
    Perturbation magnitude chosen to control attack strength while keeping changes small.
axioms (1)
  • domain assumption Neural network decision functions are approximately linear in input space near data points.
    Invoked to justify the first-order Taylor expansion used to derive the adversarial perturbation direction.

pith-pipeline@v0.9.0 · 5422 in / 1085 out tokens · 50653 ms · 2026-05-11T04:54:57.437273+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Adversarially Robust Approximate Furthest Neighbor

    cs.DS 2026-05 unverdicted novelty 8.0

    First adversarially robust data structure for c-approximate furthest neighbor search with query time matching the best known oblivious results for many parameter regimes.

  2. REALISTA: Realistic Latent Adversarial Attacks that Elicit LLM Hallucinations

    cs.CL 2026-05 unverdicted novelty 8.0

    REALISTA optimizes continuous combinations of valid editing directions in latent space to produce realistic adversarial prompts that elicit hallucinations more effectively than prior methods, including on large reason...

  3. Online Learning-to-Defer with Varying Experts

    stat.ML 2026-05 unverdicted novelty 8.0

    Presents the first online learning-to-defer algorithm with regret bounds O((n + n_e) T^{2/3}) generally and O((n + n_e) sqrt(T)) under low noise for multiclass classification with varying experts.

  4. On the Generation and Mitigation of Harmful Geometry in Image-to-3D Models

    cs.CR 2026-05 conditional novelty 8.0

    Image-to-3D models successfully generate harmful geometries in most cases with under 0.3% caught by commercial filters; existing safeguards are weak but a stacked defense cuts harmful outputs to under 1% at 11% false-...

  5. Local LMO: Constrained Gradient Optimization via a Local Linear Minimization Oracle

    math.OC 2026-05 unverdicted novelty 8.0

    Local LMO is a new projection-free method that achieves the convergence rates of projected gradient descent for constrained optimization by using local linear minimization oracles over small balls.

  6. Turn Your Face Into An Attack Surface: Screen Attack Using Facial Reflections in Video Conferencing

    cs.CR 2026-04 unverdicted novelty 8.0

    Facial reflections in video conferencing feeds can be processed to eavesdrop on on-screen application activities at 99.32% accuracy across real devices and environments.

  7. Uncovering and Understanding FPR Manipulation Attack in Industrial IoT Networks

    cs.CR 2026-01 unverdicted novelty 8.0

    FPR manipulation attack perturbs benign MQTT packets to flip labels to attacks in NIDS with 80-100% success, increasing SOC delays without gradient-based methods.

  8. Property-Preserving Hashing for $\ell_1$-Distance Predicates: Applications to Countering Adversarial Input Attacks

    cs.CR 2025-04 unverdicted novelty 8.0

    Introduces first PPH for ℓ1-distance predicate with O(t²) runtime to force significant noise in adversarial image attacks.

  9. Universal and Transferable Adversarial Attacks on Aligned Language Models

    cs.CL 2023-07 accept novelty 8.0

    Gradient and greedy search over token suffixes produces universal, transferable adversarial prompts that elicit objectionable outputs from aligned models including black-box commercial systems.

  10. Dataset Distillation

    cs.LG 2018-11 unverdicted novelty 8.0

    Dataset distillation creates a tiny synthetic training set that, when used with a fixed network initialization, produces models whose performance approximates that of models trained on the full original dataset.

  11. Detecting Adversarial Evasion Attacks Against Autoencoder-Based Network Intrusion Detection Systems

    cs.CR 2026-07 unverdicted novelty 7.0

    Two detectors achieve near-perfect accuracy detecting PANDA-style adversarial attacks on autoencoder NIDS using image-space error localization and packet-feature consistency checks on IoT traffic.

  12. A Classifier-Agnostic Zero-Shot Adversarial Attack Detection via CLIP

    cs.CV 2026-06 unverdicted novelty 7.0

    A^4D is a classifier- and attack-agnostic zero-shot adversarial attack detector based on CLIP embedding shifts that claims SOTA performance.

  13. A Classifier-Agnostic Zero-Shot Adversarial Attack Detection via CLIP

    cs.CV 2026-06 unverdicted novelty 7.0

    A^4D detects adversarial attacks in an attack- and classifier-agnostic way by measuring non-arbitrary shifts in CLIP embedding space from prompt-based similarity scores.

  14. Can Machines Really See Objects in Images? A Study Based on Syntactic Distance and Visual Self-Referential Instances

    cs.CV 2026-06 unverdicted novelty 7.0

    Vision models show a phase transition where accuracy on a zero syntactic-distance global-semantics task collapses to chance beyond a critical scale and does not recover with larger models or data.

  15. The Riddle Riddle: Testing Flexible Reasoning in Large Language Models and Humans

    cs.CL 2026-06 conditional novelty 7.0

    LLMs score 84.9% on genuine riddles but 50.7% on riddle riddles requiring literal answers, opposite to humans (50.5% vs 80.5%), indicating memory retrieval over flexible strategy selection.

  16. MIRAGE: Protecting against Malicious Image Editing via False Moderation

    cs.CR 2026-06 unverdicted novelty 7.0

    MIRAGE immunizes images by aligning them to policy-violating concepts in open-source moderation embedding spaces, triggering automatic refusals in commercial image editing APIs with over 88% success.

  17. MIRAGE: Protecting against Malicious Image Editing via False Moderation

    cs.CR 2026-06 unverdicted novelty 7.0

    MIRAGE immunizes images by crafting perturbations that align them with policy-violating concepts in open-source moderation models, triggering refusals in closed-source commercial image editors at over 88% success rate.

  18. Accelerated and Stable Convergence with Anchored Optimistic Method

    math.OC 2026-06 unverdicted novelty 7.0

    GOMA achieves optimal last-iterate O(1/k²) convergence in deterministic monotone Lipschitz VIs and O(1/√k) in stochastic unbounded-variance settings without variance reduction.

  19. Improving Adversarial Transferability on Vision-Language Pre-training Models via Surrogate-Specific Bias Correction

    cs.CV 2026-06 unverdicted novelty 7.0

    DeBias-Attack corrects surrogate-specific bias in adversarial gradients for VLP models by subtracting the projection from a reference branch optimized on weak-semantic images.

  20. Adversarial Robustness of Activation Steering in Large Language Models

    cs.LG 2026-06 unverdicted novelty 7.0

    First systematic test shows activation steering robustness drops sharply (up to 64%) under adversarial input perturbations across multiple extraction methods, models, and personas.

  21. Anti-Hyperspectral Anomaly Detection: A First Study on Stealthy Lipschitz-Forcing Perturbations Against Unknown Detectors

    eess.IV 2026-06 unverdicted novelty 7.0

    Develops the first AHAD method using ARAB regularization and Lipschitz-forcing perturbations to produce one energy-efficient signal that evades multiple unknown benchmark HAD detectors.

  22. RogueMerge: Robust and Unified Attacks against LLM Model Merging

    cs.CR 2026-06 unverdicted novelty 7.0

    RogueMerge is a unified attack method that jointly optimizes task vectors to succeed after merging, using stochastic min-max simulation for unknown merging settings and a Taylor-approximated DRO for prompt generalizat...

  23. RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations

    cs.SE 2026-05 unverdicted novelty 7.0

    RepoMirage uses semantics-preserving perturbations on SWE-Bench to show code agents lack repository context reasoning, with performance falling sharply on extended structure tasks, and introduces RepoAnchor as a struc...

  24. When Interpretability Becomes a Liability: Adversarial Attacks on CBM Concept Layers

    cs.LG 2026-05 unverdicted novelty 7.0

    Concept-level adversarial attacks exploit CBM interpretability on the CUB dataset, but SPECTRA raises required perturbation norm from 0.46 to over 4200 while keeping accuracy loss under 2.2%.

  25. Understanding Dynamics of Adam in Zero-Sum Games: An ODE Approach

    cs.LG 2026-05 unverdicted novelty 7.0

    Derives ODE limits of Adam-DA showing that first- and second-order momentum parameters reverse their convergence roles in zero-sum games compared to minimization, validated on GAN experiments.

  26. Interaction-Breaking Adversarial Learning Framework for Robust Multi-Agent Reinforcement Learning

    cs.LG 2026-05 unverdicted novelty 7.0

    IBAL framework constructs information-theoretic adversarial attacks on agent observations and actions to train MARL agents that remain robust to interaction disruptions and agent-missing scenarios.

  27. Escaping the Linearity Trap: Manifold Detours for Black-Box Adversarial Attacks on Singing Audio Deepfake Detection

    cs.CR 2026-05 unverdicted novelty 7.0

    MARS is a transfer-based black-box attack that uses bi-level optimization on semantic and artifact anchors to escape the linearity trap and improve attack success rates on SSL-SVDD by up to 36%.

  28. AIM: Adversarial Information Masking for Faithfulness Evaluation of Saliency Maps

    cs.LG 2026-05 unverdicted novelty 7.0

    AIM is a new saliency-guided adversarial feature replacement method to evaluate faithfulness of saliency maps and reliability of masking operators on image, audio, and EEG tasks.

  29. $\alpha$-TCAV: A Unified Framework for Testing with Concept Activation Vectors

    stat.ML 2026-05 unverdicted novelty 7.0

    α-TCAV replaces TCAV's hard indicator with a tunable smooth function to create a unified probabilistic framework with lower variance and guidance for parameter choice or Bayes-optimal scoring.

  30. Quantitative Linear Logic for Neuro-Symbolic Learning and Verification

    cs.LO 2026-05 unverdicted novelty 7.0

    QLL is a novel logic for neuro-symbolic learning that uses ML-native operations (sum, log-sum-exp) on logits to embed constraints, satisfying most linear logic properties and showing stronger correlation between empir...

  31. Online Learning-to-Defer with Varying Experts

    stat.ML 2026-05 unverdicted novelty 7.0

    Presents first online L2D algorithm for multiclass classification with bandit feedback and varying experts, achieving O((n+n_e)T^{2/3}) regret generally and O((n+n_e)√T) under low noise.

  32. Online Learning-to-Defer with Varying Experts

    stat.ML 2026-05 unverdicted novelty 7.0

    Presents the first online Learning-to-Defer algorithm achieving regret O((n + n_e) T^{2/3}) generally and O((n + n_e) sqrt(T)) under low noise for multiclass classification with varying experts.

  33. A neurosymbolic Approach with Epistemic Deep Learning for Hierarchical Image Classification

    cs.CV 2026-05 unverdicted novelty 7.0

    A neurosymbolic model augments Swin Transformers with focal sets and fuzzy logic to produce calibrated hierarchical image classifications that respect logical constraints.

  34. TARO: Temporal Adversarial Rectification Optimization Using Diffusion Models as Purifiers

    cs.LG 2026-05 unverdicted novelty 7.0

    TARO builds a temporally guided score prior from high-noise and low-noise diffusion views to purify adversarial examples more robustly than uniform timestep methods.

  35. Inference Time Causal Probing in LLMs

    cs.AI 2026-05 unverdicted novelty 7.0

    HDMI is a new probe-free technique that steers LLM hidden states via margin objectives to achieve more reliable causal interventions than prior probe-based methods on standard benchmarks.

  36. Streaming Adversarial Robustness in Fuzzy ARTMAP: Mechanism-Aligned Evaluation, Progressive Training, and Interpretable Diagnostics

    cs.LG 2026-05 conditional novelty 7.0

    Fuzzy ARTMAP models are highly vulnerable to a new white-box attack aligned with their category competition, but progressive selective training yields stronger replay-free robustness than offline adversarial training ...

  37. Empirical Evidence for Simply Connected Decision Regions in Image Classifiers

    cs.CV 2026-05 unverdicted novelty 7.0

    Empirical tests with quad-mesh filling indicate that decision regions in modern image classifiers are simply connected.

  38. Sparse Tokens Suffice: Jailbreaking Audio Language Models via Token-Aware Gradient Optimization

    cs.CR 2026-05 conditional novelty 7.0

    TAGO performs sparse jailbreak optimization on audio LMs by retaining only high-gradient-energy tokens, preserving near-full ASR at 25% retention across three models.

  39. Sparse Tokens Suffice: Jailbreaking Audio Language Models via Token-Aware Gradient Optimization

    cs.CR 2026-05 conditional novelty 7.0

    Sparse selection of high-gradient-energy audio tokens suffices for effective jailbreaking of audio language models with minimal drop in attack success rate.

  40. Minimum Specification Perturbation: Robustness as Distance-to-Falsification in Causal Inference

    stat.ME 2026-05 unverdicted novelty 7.0

    MSP quantifies the minimum changes to analyst choices required to falsify a causal claim by making its confidence interval contain zero, providing information orthogonal to dispersion-based robustness summaries.

  41. Decision Boundary-aware Generation for Long-tailed Learning

    cs.CV 2026-05 unverdicted novelty 7.0

    DBG mitigates boundary overlap in long-tailed learning by generating near-boundary samples, leading to better tail class accuracy and more separable decision spaces.

  42. Quantum Interval Bound Propagation for Certified Training of Quantum Neural Networks

    quant-ph 2026-05 unverdicted novelty 7.0

    QIBP adapts interval bound propagation to quantum neural networks for certified adversarial robustness via interval and affine arithmetic implementations.

  43. From Local to Global to Mechanistic: An iERF-Centered Unified Framework for Interpreting Vision Models

    cs.CV 2026-05 unverdicted novelty 7.0

    An iERF-centric framework unifies local, global, and mechanistic interpretability in vision models via SRD for saliency, CAFE for concept anchoring, and ICAT for interlayer attribution.

  44. Low Rank Adaptation for Adversarial Perturbation

    cs.LG 2026-04 unverdicted novelty 7.0

    Adversarial perturbations possess an inherently low-rank structure that enables more efficient and effective black-box adversarial attacks via subspace projection.

  45. A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Framework

    cs.CR 2026-04 unverdicted novelty 7.0

    A new 7x4 taxonomy organizes agentic AI security threats by architectural layer and persistence timescale, revealing under-explored upper layers and missing defenses after surveying 116 papers.

  46. Benign Overfitting in Adversarial Training for Vision Transformers

    cs.LG 2026-04 unverdicted novelty 7.0

    Adversarial training on simplified Vision Transformers achieves benign overfitting with near-zero robust loss and generalization error when signal-to-noise ratio and perturbation budget meet specific conditions.

  47. Local Linearity of LLMs Enables Activation Steering via Model-Based Linear Optimal Control

    cs.LG 2026-04 conditional novelty 7.0

    Local linearity of LLM layers enables LQR-based closed-loop activation steering with theoretical tracking guarantees.

  48. Duality for the Adversarial Total Variation

    math.AP 2026-04 unverdicted novelty 7.0

    Duality techniques produce a dual representation and subdifferential characterization for the nonlocal total variation functional arising in adversarial training.

  49. Feature-level analysis and adversarial transfer in rotationally equivariant quantum machine learning

    quant-ph 2026-04 unverdicted novelty 7.0

    Rotationally equivariant quantum models can rely on vulnerable invariant statistics such as ring-averaged intensities, leaving them susceptible to classical transfer attacks, but suppressing the associated symmetry se...

  50. Physically-Induced Atmospheric Adversarial Perturbations: Enhancing Transferability and Robustness in Remote Sensing Image Classification

    cs.CV 2026-04 unverdicted novelty 7.0

    FogFool creates fog-based adversarial perturbations using Perlin noise optimization to achieve high black-box transferability (83.74% TASR) and robustness to defenses in remote sensing classification.

  51. Understanding and Improving Continuous Adversarial Training for LLMs via In-context Learning Theory

    cs.LG 2026-04 unverdicted novelty 7.0

    Continuous adversarial training in the embedding space produces a robust generalization bound for linear transformers that decreases with perturbation radius, tied to singular values of the embedding matrix, and motiv...

  52. Learning Robustness at Test-Time from a Non-Robust Teacher

    cs.CV 2026-04 unverdicted novelty 7.0

    A test-time adaptation framework anchors adversarial training to a non-robust teacher's predictions, yielding more stable optimization and better robustness-accuracy trade-offs than standard self-consistency methods.

  53. Efficient Unlearning through Maximizing Relearning Convergence Delay

    cs.LG 2026-04 unverdicted novelty 7.0

    The Influence Eliminating Unlearning framework maximizes relearning convergence delay via weight decay and noise injection to remove the influence of a forgetting set while preserving accuracy on retained data.

  54. On the Decompositionality of Neural Networks

    cs.LO 2026-04 unverdicted novelty 7.0

    Neural decompositionality is defined via decision-boundary semantic preservation, and language transformers largely satisfy it under SAVED while vision models often do not.

  55. Can Drift-Adaptive Malware Detectors Be Made Robust? Attacks and Defenses Under White-Box and Black-Box Threats

    cs.CR 2026-04 unverdicted novelty 7.0

    A fine-tuning framework reduces PGD attack success on AdvDA detectors from 100% to 3.2% and MalGuise from 13% to 5.1%, but optimal training strategies differ by threat model and robustness does not transfer across them.

  56. Revealing Physical-World Semantic Vulnerabilities: Universal Adversarial Patches for Infrared Vision-Language Models

    cs.CV 2026-04 unverdicted novelty 7.0

    UCGP is a universal physical adversarial patch that compromises cross-modal semantic alignment in IR-VLMs through curved-grid parameterization and representation-space disruption.

  57. Hidden Reliability Risks in Large Language Models: Systematic Identification of Precision-Induced Output Disagreements

    cs.AI 2026-04 unverdicted novelty 7.0

    PrecisionDiff is a differential testing framework that uncovers widespread precision-induced behavioral disagreements in aligned LLMs, including safety-critical jailbreak divergences across precision formats.

  58. A Speculative GLRT-Backed ApproachRobust Deep Learning-Based Array Processing

    eess.SP 2025-12 unverdicted novelty 7.0

    A speculative DL classifier validated by GLRT on spatially robust second-order statistics provides adversarially resilient array processing.

  59. Adversarial Robustness in One-Stage Learning-to-Defer

    stat.ML 2025-10 unverdicted novelty 7.0

    Develops the first adversarial robustness framework for one-stage learning-to-defer, including cost-sensitive surrogate losses and theoretical consistency guarantees for classification and regression.

  60. SCOOTER: A Human Evaluation Framework for Unrestricted Adversarial Examples

    cs.CV 2025-07 conditional novelty 7.0

    SCOOTER supplies best-practice guidelines, open tools, and a 3K-image benchmark with 34K+ human ratings showing that six tested unrestricted attacks produce images humans can detect as fake.