pith. sign in

arxiv: 2605.02495 · v2 · pith:TA5KJQNTnew · submitted 2026-05-04 · 💻 cs.LG · cs.AI· stat.ML

Efficient Preference Poisoning Attack on Offline RLHF

Chenye Yang , Weiyu Xu , Lifeng Lai This is my paper

Pith reviewed 2026-06-30 23:51 UTC · model grok-4.3

classification 💻 cs.LG cs.AIstat.ML
keywords preference poisoningoffline RLHFDPOlabel flip attacksparse approximationlattice reductionmatching pursuitgradient shift
0
0 comments X

The pith

A single preference label flip creates a parameter-independent shift in the DPO gradient, turning targeted poisoning into a solvable binary sparse approximation problem.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that flipping one label in a DPO preference dataset shifts the gradient in a direction that does not depend on the current policy parameters. This independence lets the authors recast the choice of which labels to flip as a structured binary sparse approximation task over a gradient dictionary. They introduce BAL-A, which embeds the problem in a binary-aware lattice and uses LLL reduction plus Babai's nearest-plane method, and BMP-A, which adapts binary matching pursuit and supplies coherence-based recovery guarantees. A reader would care because the result shows how small, structured changes to offline preference data can steer the learned policy with provable efficiency.

Core claim

Flipping one preference label induces a parameter-independent shift in the DPO gradient. This key property converts the targeted poisoning problem into a structured binary sparse approximation problem. The authors solve it with Binary-Aware Lattice Attack (BAL-A) that uses LLL reduction and Babai's nearest plane algorithm under sufficient conditions enforcing binary coefficients and minimum-flip recovery, and with Binary Matching Pursuit Attack (BMP-A) that yields coherence-based recovery guarantees and robustness certificates for K-flip budgets.

What carries the argument

The parameter-independent gradient shift from a single preference label flip, which reduces targeted poisoning to a binary sparse approximation problem solved over the non-normalized gradient dictionary.

If this is right

  • Targeted poisoning succeeds with few flips whenever the gradient dictionary satisfies the lattice or coherence conditions.
  • BAL-A recovers the exact minimum-flip set when the lattice embedding meets the stated sufficient conditions.
  • BMP-A certifies when no attack with a given K-flip budget can reach the target direction.
  • Attack success is controlled by the geometry, specifically the coherence, of the gradient dictionary.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Defenses could aim to increase dictionary coherence or add controlled noise to preference labels to raise the minimum number of flips needed.
  • The same independence property may let similar sparse-recovery attacks apply to other log-linear preference objectives beyond DPO.
  • Data curators might need integrity checks that detect structured small changes rather than random noise.
  • The framing suggests studying whether other offline RLHF pipelines admit comparable parameter-independent gradient shifts.

Load-bearing premise

The DPO objective is strictly log-linear in the policy parameters so the gradient shift after any single label flip stays independent of the current parameter vector.

What would settle it

An experiment that measures the DPO gradient change after a label flip and finds that the direction or magnitude varies with the policy parameters would falsify the independence property.

Figures

Figures reproduced from arXiv: 2605.02495 by Chenye Yang, Lifeng Lai, Weiyu Xu.

Figure 2
Figure 2. Figure 2: TPR of BMP-A on synthetic V as a function of K⋆ view at source ↗
Figure 1
Figure 1. Figure 1: TPR of BAL-A on synthetic V as a function of M. 6.2.2. VALIDATE BMP-A THEORY ON SYNTHETIC V [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 1
Figure 1. Figure 1: TPR of BAL-A on synthetic V as a function of M view at source ↗
Figure 3
Figure 3. Figure 3: gives the complementary clean-vs-attacked diag￾nostic on the common feasible SHP case. In this matched case, BAL-A and BMP-A recover the same constructed flip pattern exactly (TP = 7, FP = 0, FN = 0), so their at￾tacked subsets coincide. This also allows a direct runtime check: BAL-A takes 0.6865 seconds, while BMP-A takes 1.37 × 10−4 seconds. We include this comparison only as a matched diagnostic, and th… view at source ↗
Figure 3
Figure 3. Figure 3: True positive rate of BAL-A on V from SHP as a function of M view at source ↗
Figure 2
Figure 2. Figure 2: TPR of BMP-A on synthetic V as a function of K⋆ [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: ℓ2 distance between learned parameters and ℓ1 distance between learned policies, comparing training on the BAL-A attacked subset D˜(ˆx) versus training on the ground-truth attacked subset D˜(x ⋆ ), as a function of M view at source ↗
Figure 5
Figure 5. Figure 5: Histogram of pairwise normalized correlations for two subsets of SHP: a random subset and a low-coherence subset. y = V x⋆ , then run BMP-A with tolerance ε = 10−3 up to budget tmax = 15 with 200 trials. The low-coherence subset yields consistently higher TPR as the budget increases and drives the residual down faster, often reaching a near-zero residual around K⋆ . In contrast, on the random subset BMP-A … view at source ↗
Figure 5
Figure 5. Figure 5: Attack-vs-groundtruth diagnostic for BAL-A: ℓ2 distance between learned parameters and ℓ1 distance between learned policies, comparing training on the BAL-A attacked subset D˜(ˆx) versus training on the ground-truth attacked subset D˜(x ⋆ ), as a function of M [PITH_FULL_IMAGE:figures/full_fig_p022_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: True positive rate and residual of BMP-A on V from different subset of SHP as a function of budget K. and similarly for µ = πθµ . Therefore, log πθ(ai | si) µ(ai | si) − log πθ(a ′ i | si) µ(a ′ i | si) = ψ(si , ai) ⊤θ − ψ(si , ai) ⊤θµ − ψ(si , a′ i ) ⊤θ + ψ(si , a′ i ) ⊤θµ = ∆ψ ⊤ i (θ − θµ), where ∆ψi := ψ(si , ai) − ψ(si , a′ i ) ∈ R d . Plugging into (1) gives per-sample loss ℓi(θ) = − log σ view at source ↗
Figure 7
Figure 7. Figure 7: ℓ2 distance between learned parameters and ℓ1 distance between learned policies, comparing training on the BMP-A attacked D˜(ˆx) versus training on the ground-truth attacked D˜(x ⋆ ), as a function of budget K. Using the sigmoid identity σ(x) = 1 − σ(−x), we have ∆gi(θ) = oiβ∆ψi =: ∆gi . Here we see that the gradient shift caused by flipping the label of one sample is a constant vector, independent of the … view at source ↗
Figure 8
Figure 8. Figure 8: Attack-vs-groundtruth diagnostic for BMP-A: ℓ2 distance between learned parameters and ℓ1 distance between learned policies, comparing training on the BMP-A attacked D˜(ˆx) versus training on the ground-truth attacked D˜(x ⋆ ), as a function of budget K. and similarly for µ = πθµ . Therefore, log πθ(ai | si) µ(ai | si) − log πθ(a ′ i | si) µ(a ′ i | si) = ψ(si , ai) ⊤θ − ψ(si , ai) ⊤θµ − ψ(si , a′ i ) ⊤θ +… view at source ↗
read the original abstract

Offline Reinforcement Learning from Human Feedback (RLHF) pipelines such as Direct Preference Optimization (DPO) train on a pre-collected preference dataset, which makes them vulnerable to preference poisoning attack. We study label flip attacks against log-linear DPO. We first illustrate that flipping one preference label induces a parameter-independent shift in the DPO gradient. Using this key property, we can then convert the targeted poisoning problem into a structured binary sparse approximation problem. To solve this problem, we develop two attack methods: Binary-Aware Lattice Attack (BAL-A) and Binary Matching Pursuit Attack (BMP-A). BAL-A embeds the binary flip selection problem into a binary-aware lattice and applies Lenstra-Lenstra-Lov\'asz reduction and Babai's nearest plane algorithm; we provide sufficient conditions that enforce binary coefficients and recover the minimum-flip objective. BMP-A adapts binary matching pursuit to our non-normalized gradient dictionary and yields coherence-based recovery guarantees and robustness (impossibility) certificates for $K$-flip budgets. Experiments on synthetic dictionaries and the Stanford Human Preferences dataset validate the theory and highlight how dictionary geometry governs attack success.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 3 minor

Summary. The manuscript analyzes label-flip poisoning attacks on offline DPO for log-linear policies. It establishes that a single preference-label flip produces a parameter-independent gradient shift of exactly βϕ (where ϕ is the feature difference), reducing targeted poisoning to a binary sparse approximation problem over the dictionary of ϕ vectors. Two solvers are introduced: BAL-A, which embeds the problem in a binary-aware lattice and applies LLL reduction plus Babai’s nearest-plane algorithm with sufficient conditions for exact binary recovery, and BMP-A, which adapts binary matching pursuit with coherence-based recovery guarantees and K-flip impossibility certificates. Experiments on synthetic dictionaries and the Stanford Human Preferences dataset confirm the geometry dependence of attack success.

Significance. If the gradient-shift property and recovery conditions hold, the work supplies the first explicit, parameter-free reduction of a practical RLHF poisoning task to a standard sparse-approximation instance together with two algorithms that carry provable guarantees. This is a concrete contribution to the security analysis of offline RLHF pipelines, showing how dictionary coherence directly controls attack feasibility.

major comments (3)
  1. [§3] §3 (gradient-shift derivation): the explicit differentiation showing that the sigmoid terms cancel to leave exactly βϕ must appear in the main text with the reference-model term retained; the abstract alone does not constitute the required derivation.
  2. [Theorem 1] Theorem 1 (BAL-A recovery): the stated sufficient conditions for the lattice-reduced basis to force binary coefficients must be accompanied by an explicit bound on the Babai approximation factor relative to the minimum distance of the lattice; without this the claim that the algorithm recovers the minimum-flip solution is not fully supported.
  3. [BMP-A section] BMP-A coherence guarantee: the recovery threshold is expressed in terms of the mutual coherence of the (non-normalized) gradient dictionary; the paper must verify that this coherence is computed after the β scaling and reference-model subtraction, otherwise the stated bound does not apply to the actual attack instance.
minor comments (3)
  1. Notation for the feature-difference vectors ϕ should be introduced once and used uniformly; the abstract’s “dictionary of ϕ vectors” and the main-text definition must match exactly.
  2. [Experiments] In the Stanford Human Preferences experiments, report the measured coherence value of the empirical dictionary so that readers can directly compare it with the theoretical recovery thresholds.
  3. [Figures] Figure captions should state whether the plotted success rates are averaged over multiple random target pairs or correspond to a single fixed target.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the careful reading and constructive comments on our manuscript. We address each major comment below and will incorporate the necessary revisions.

read point-by-point responses

  1. Referee: [§3] §3 (gradient-shift derivation): the explicit differentiation showing that the sigmoid terms cancel to leave exactly βϕ must appear in the main text with the reference-model term retained; the abstract alone does not constitute the required derivation.

    Authors: We agree that the full derivation should be presented explicitly in the main text. We will expand Section 3 to include the complete differentiation steps, retaining the reference-model term throughout. revision: yes

  2. Referee: [Theorem 1] Theorem 1 (BAL-A recovery): the stated sufficient conditions for the lattice-reduced basis to force binary coefficients must be accompanied by an explicit bound on the Babai approximation factor relative to the minimum distance of the lattice; without this the claim that the algorithm recovers the minimum-flip solution is not fully supported.

    Authors: We acknowledge the need for an explicit bound relating the Babai approximation factor to the minimum lattice distance. We will add this bound to the statement and proof of Theorem 1 in the revised manuscript. revision: yes

  3. Referee: [BMP-A section] BMP-A coherence guarantee: the recovery threshold is expressed in terms of the mutual coherence of the (non-normalized) gradient dictionary; the paper must verify that this coherence is computed after the β scaling and reference-model subtraction, otherwise the stated bound does not apply to the actual attack instance.

    Authors: We will add an explicit verification in the BMP-A section confirming that the mutual coherence is computed on the dictionary after β scaling and reference-model subtraction, ensuring the recovery guarantees apply directly to the attack setting. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation is mathematically self-contained

full rationale

The central property (parameter-independent gradient shift after one label flip) is obtained by explicit differentiation of the DPO loss, yielding a constant difference βϕ independent of θ. This identity is used to reduce the attack to a standard binary sparse approximation problem, which is then solved by known algorithms (lattice reduction + Babai, matching pursuit) with coherence guarantees. No step reduces to a fitted parameter renamed as prediction, a self-definitional loop, or a load-bearing self-citation whose content is unverified. The paper is self-contained against external benchmarks (synthetic dictionaries, Stanford Human Preferences dataset) and the derivation chain does not collapse to its inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the log-linear form of DPO and the parameter-independence of the single-flip gradient shift; both are domain assumptions rather than derived results.

axioms (1)
  • domain assumption DPO loss is log-linear in the policy parameters
    Explicitly invoked when the abstract states 'log-linear DPO' and the parameter-independent gradient shift.

pith-pipeline@v0.9.1-grok · 5727 in / 1205 out tokens · 24794 ms · 2026-06-30T23:51:24.684992+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

4 extracted references · 1 canonical work pages

  1. [1]

    arXiv preprint arXiv:2409.15360 , archivePrefix =

    URL https://www.noahsd.com/mini_ lattices/05__babai.pdf. Sui, Y ., Zoghi, M., Hofmann, K., and Yue, Y . Advance- ments in dueling bandits. InProceedings of the Inter- national Joint Conference on Artificial Intelligence, pp. 5502–5510, Stockholm, Sweden, Jul. 2018. Sun, J., Zhang, T., Xie, X., Ma, L., Zheng, Y ., Chen, K., and Liu, Y . Stealthy and effici...

    work page arXiv 2018

  2. [2]

    minimum flips).The CVP-style formulation focuses on minimizing the residual minx∈{0,1}n ∥V x+g †∥2, whereas our flip attack is aminimum-cardinalityproblem: minx∈{0,1}n 1⊤x

    Objective mismatch (minimum residual vs. minimum flips).The CVP-style formulation focuses on minimizing the residual minx∈{0,1}n ∥V x+g †∥2, whereas our flip attack is aminimum-cardinalityproblem: minx∈{0,1}n 1⊤x. Even when an exact attack is feasible (so the residual is zero), residual minimization alone does not distinguish among multiple feasible solut...

  3. [3]

    In our setting, each data point can be flipped at most once, so the admissible coefficients must lie in{0,1}

    Unbounded integer coefficients.In (9), the coefficients zi can be arbitrary integers, corresponding to “using” the same flip-effect vi multiple times. In our setting, each data point can be flipped at most once, so the admissible coefficients must lie in{0,1}. Large integer coefficients are not meaningful as physical flip patterns

  4. [4]

    No guaranteed truncation.A common workaround is to truncate an integer solution to the binary hypercube, e.g., xi =1{z i ≥1} . This nonlinear projection can destroy the relationship between ∥V z+g †∥2 and ∥V x+g †∥2, and there is no guarantee that a near-optimal integer solution leads to a near-optimal binary solution. E. BAL-A: Full Binary-Aware Lattice ...

    1982