REFLECTOR: Internalizing Step-wise Reflection against Indirect Jailbreak

Bo Zou; Chaochao Lu; Chao Yang; Jiachen Ma; Jiawen Zhang; Xiangtian Li

REVIEW 2 major objections 1 minor 8 references

Reflector internalizes step-wise self-reflection via two-stage training to defend LLMs against indirect jailbreaks while improving task performance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 17:53 UTC pith:46ZH5J3D

load-bearing objection Reflector describes a two-stage SFT-then-RL pipeline to internalize step-wise reflection for indirect jailbreaks, but the abstract supplies no experimental details, baselines, or reward construction to support the >90% DSR and utility claims. the 2 major comments →

arxiv 2605.20654 v2 pith:46ZH5J3D submitted 2026-05-20 cs.LG cs.AI

REFLECTOR: Internalizing Step-wise Reflection against Indirect Jailbreak

Jiachen Ma , Jiawen Zhang , Xiangtian Li , Bo Zou , Chaochao Lu , Chao Yang This is my paper

classification cs.LG cs.AI

keywords jailbreak defenseself-reflectionLLM safetyreinforcement learningsupervised fine-tuningindirect attackstrajectory safety

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a two-stage method that first creates reflection data through teacher-guided supervised fine-tuning and then refines it with reinforcement learning using outcome-driven and reward-validity signals. This internalizes safety checks directly into the model's generation trajectory rather than relying on external filters. If the approach works as described, models would resist multi-step attacks that exploit internal processes while also showing gains on math and knowledge tasks. The core motivation is that surface-level alignment leaves models open to sophisticated indirect exploits.

Core claim

Reflector establishes structured reflection patterns through teacher-guided SFT and then uses RL with outcome-driven and reward-validity supervision to produce autonomous self-reflection, yielding defense success rates above 90 percent against complex indirect attacks, robust generalization across threat scenarios, and utility gains including a 5.85 percent improvement on GSM8K plus better results on knowledge benchmarks.

What carries the argument

The two-stage framework of teacher-guided SFT to establish reflection patterns followed by RL with outcome-driven and reward-validity supervision.

Load-bearing premise

The two-stage training will produce autonomous self-reflection that generalizes to new attacks without reducing core model capabilities or creating fresh failure modes.

What would settle it

Testing the trained model on a fresh collection of indirect jailbreak prompts withheld from both training stages and checking whether defense rates stay above 90 percent while task accuracy on GSM8K does not drop below the reported gain.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Models gain defense against multi-step indirect attacks without added inference cost.
Safety improvements coincide with higher accuracy on math and knowledge tasks rather than trade-offs.
Trajectory-level reflection scales to diverse threat types beyond the evaluated scenarios.
The internalized process avoids the overhead of separate safety modules at deployment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same reflection mechanism might reduce other generation errors such as factual inconsistencies if the reward signals were adjusted accordingly.
Extending the RL stage to include process-level rewards could further strengthen resistance to attacks that target early generation steps.
If reflection becomes a default training component, future models might require less post-hoc safety patching across applications.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Reflector describes a two-stage SFT-then-RL pipeline to internalize step-wise reflection for indirect jailbreaks, but the abstract supplies no experimental details, baselines, or reward construction to support the >90% DSR and utility claims.

read the letter

The main point is that this paper outlines a concrete two-stage process: teacher-guided SFT to install reflection patterns, followed by RL using outcome-driven rewards plus reward-validity supervision, with the goal of making the model defend against indirect, multi-step jailbreaks through internal trajectory reflection rather than surface rules.

The approach is a straightforward extension of existing reflection and RLHF techniques applied to a real weakness in current alignment. Targeting the generation process itself and claiming both strong defense and a utility lift (5.85% on GSM8K) is at least directionally sensible if the RL stage actually produces autonomous behavior.

The soft spots are substantial and center on missing evidence. The abstract states the performance numbers but gives no description of the attack datasets, baselines, statistical tests, held-out scenarios, or how the reward-validity signal is built to prevent hacking. Without ablations on the RL stage or checks for new failure modes like over-reflection, it is impossible to tell whether the claimed generalization holds or whether the model simply satisfies the training signals. The stress-test concern about autonomous reflection without degrading capabilities or introducing side effects is not addressed in the provided text.

This is aimed at researchers working on LLM safety and robustness. Someone already experimenting with reflection or RL for alignment might pick up the pipeline structure as a starting point, but only after the full methods and results sections are available to evaluate.

If the complete paper contains reproducible experiments, clear reward definitions, and proper comparisons, it would merit peer review. Based on the abstract alone the claims are too underspecified to assess.

Referee Report

2 major / 1 minor

Summary. The manuscript proposes REFLECTOR, a two-stage framework for defending LLMs against indirect jailbreak attacks by internalizing step-wise self-reflection. The first stage performs teacher-guided supervised fine-tuning (SFT) on high-quality reflection data to install structured patterns; the second stage applies reinforcement learning (RL) using outcome-driven rewards together with reward-validity supervision to convert those patterns into autonomous behavior. The central claims are that the resulting model attains Defense Success Rates (DSR) exceeding 90 % on complex indirect attacks, generalizes across threat scenarios, and simultaneously improves utility (5.85 % gain on GSM8K plus gains on knowledge-intensive benchmarks) without appreciable computational overhead.

Significance. If the empirical results and the two-stage training procedure can be verified, the work would be significant: it moves safety from external filters or surface-level alignment to an internalized, trajectory-level mechanism and reports simultaneous safety and utility gains. Such an approach, if reproducible, would address a recognized limitation of current alignment techniques and could influence subsequent research on self-reflective generation.

major comments (2)

[Abstract] Abstract: the manuscript states concrete quantitative results (DSR >90 %, 5.85 % GSM8K gain, improved knowledge benchmarks) yet supplies no experimental setup, attack datasets, baselines, number of trials, or statistical tests. Because these numbers constitute the primary evidence for the central claim, their unverifiability is load-bearing.
The description of the RL stage asserts that outcome-driven plus reward-validity supervision produces autonomous reflection that generalizes to unseen attacks without new failure modes or capability degradation, but the manuscript provides neither the concrete reward construction nor ablations isolating the RL stage from the preceding SFT stage. This assumption is load-bearing for the generalization and utility claims.

minor comments (1)

[Abstract] The abstract would be clearer if it briefly indicated the base model(s) and scale at which the reported numbers were obtained.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below and will incorporate revisions to improve verifiability and explicitness while preserving the manuscript's contributions.

read point-by-point responses

Referee: [Abstract] Abstract: the manuscript states concrete quantitative results (DSR >90 %, 5.85 % GSM8K gain, improved knowledge benchmarks) yet supplies no experimental setup, attack datasets, baselines, number of trials, or statistical tests. Because these numbers constitute the primary evidence for the central claim, their unverifiability is load-bearing.

Authors: We agree that the abstract's brevity leaves the quantitative claims less immediately verifiable. In the revision we will insert a concise clause summarizing the primary attack datasets, baselines, and evaluation protocol. Complete experimental details, including trial counts and statistical reporting, already appear in Sections 4 and 5; the abstract change will simply surface them at the front. revision: partial
Referee: The description of the RL stage asserts that outcome-driven plus reward-validity supervision produces autonomous reflection that generalizes to unseen attacks without new failure modes or capability degradation, but the manuscript provides neither the concrete reward construction nor ablations isolating the RL stage from the preceding SFT stage. This assumption is load-bearing for the generalization and utility claims.

Authors: We acknowledge that the reward formulation and isolating ablations merit more explicit treatment. The revised manuscript will add the precise mathematical definitions of the outcome-driven reward (task-success indicator plus validity verifier score) and the auxiliary validity supervision loss, together with a dedicated ablation table that isolates the RL stage from the SFT stage and quantifies its contribution to generalization and utility. revision: yes

Circularity Check

0 steps flagged

No derivation chain; empirical results only

full rationale

The manuscript describes a two-stage training procedure (teacher-guided SFT followed by RL) and reports measured Defense Success Rates and benchmark gains. No equations, uniqueness theorems, fitted parameters renamed as predictions, or self-citation load-bearing steps appear. All central claims are presented as direct experimental outcomes on held-out attacks and utility tasks, with no reduction of any result to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review yields no identifiable free parameters, axioms, or invented entities; all technical details are absent.

pith-pipeline@v0.9.1-grok · 5730 in / 1075 out tokens · 35942 ms · 2026-06-30T17:53:09.117063+00:00 · methodology

0 comments

read the original abstract

While Large Language Models (LLMs) demonstrate remarkable capabilities, they remain susceptible to sophisticated, multi-step jailbreak attacks that circumvent conventional surface-level safety alignment by exploiting the internal generation process. To address these vulnerabilities, we propose Reflector, a principled two-stage framework that internalizes self-reflection within the generation trajectory. Reflector first leverages teacher-guided generation to produce high-quality reflection data for supervised fine-tuning (SFT), establishing structured reflection patterns. It subsequently uses Reinforcement Learning (RL) with outcome-driven and reward-validity supervision to instill robust, autonomous self-reflection capabilities. Empirical results show that Reflector achieves Defense Success Rates (DSR) exceeding 90% against complex indirect attacks while generalizing robustly across diverse threat scenarios. Notably, the framework enhances both task-specific and general utility, yielding a 5.85% gain on GSM8K alongside improved performance on knowledge-intensive benchmarks. By internalizing trajectory-level safety, Reflector overcomes the fundamental limitations of surface alignment without significant computational overhead, offering an efficient and scalable solution for the development of safe and capable LLMs.

Figures

Figures reproduced from arXiv: 2605.20654 by Bo Zou, Chaochao Lu, Chao Yang, Jiachen Ma, Jiawen Zhang, Xiangtian Li.

**Figure 1.** Figure 1: Correlation between the position of the first occurrence of harmful tokens and the attack success rate (ASR). While direct jailbreaks (blue) manifest immediately, indirect attacks (red) exhibit a stealthy latency, with harmful content emerging only after 20 tokens. This delay enables malicious intent to bypass surfacelevel safety alignment, leading to significantly higher ASRs than direct attacks. solvin… view at source ↗

**Figure 2.** Figure 2: The framework of REFLECTOR. In Stage 1 (SFT), the model learns the “search-and-recovery” reflection pattern from teacher-guided data. In Stage 2 (RL), the model undergoes self-improvement via GDPO, guided by a hybrid reward function that jointly optimizes for final response safety (rsafety) and the validity of the reflection process (rreflect). disrupt the generation process. Thus, learning when and how to… view at source ↗

**Figure 3.** Figure 3: Impact of safety data scaling. (a) Increasing safety data yields initial gains but ultimately degrades general performance due to over-alignment. (b) Higher safety ratios consistently strengthen reflective defenses against overtly harmful queries. model does not produce explicit reflection markers without prior SFT, we prepend each query with an instruction that specifies the required reflection format du… view at source ↗

Review history (2 revisions) →

discussion (0)

Reference graph

Works this paper leans on

8 extracted references

[1]

gold standard

to generate structured reflection-continuation pairs. These trajectories serve as the “gold standard” for internalizing safety reasoning and establishing the desired self-reflection policy. B.1. Synthesis Pipeline and Data Quality The construction of a complete trajectory ˜τ= (ybefore, z, yafter) is designed to simulate the emergence of a safety signal du...
[2]

, T} to existing model outputs

Strategic Context Truncation:To ensure the model learns to trigger reflection at various stages of response generation, we apply a random truncation strategy n∼ U {1, . . . , T} to existing model outputs. This creates a diverse set of ybefore prefixes, ranging from early-stage intent alignment to late-stage detail generation
[3]

post-mortem

Structured Reflection Generation (z):Using theSelf-Critique & Reflection Template, the teacher model performs a “post-mortem” analysis of the truncated prefix. By framing the teacher as a student reflecting on its own mistakes, we generatez reflect to identify the precise ethical breach andz explore to chart a safe path forward
[4]

search-and-recovery

Reasoning-Conditioned Continuation ( yafter):The teacher generates yafter by strictly adhering to the guidance provided in z. This stage ensures that the final output is not just a refusal, but a logically consistent continuation that follows the “search-and-recovery” logic established in the reflection phase. To maintain high data fidelity, we perform a ...

2025
[5]

As reported, this model demonstrates high robustness, outperforming GPT-4 by approximately 5% in agreement rates with human judgments on manually labeled validation sets

Discriminative Detector ( Cdisc).We utilize the HarmBench classifier (Mazeika et al., 2024), which is fine-tuned on Llama-2-13B. As reported, this model demonstrates high robustness, outperforming GPT-4 by approximately 5% in agreement rates with human judgments on manually labeled validation sets. It provides a binary verdict: Cdisc(y) = ( 1if Harmless 0...

2024
[6]

Generative Detector ( Jgen).To capture nuance missed by binary classification, we concurrently deploy GPT-OSS- 120B (Agarwal et al., 2025) as a generative safety detector. This model evaluates the semantic content of the response trajectory and categorizes it into one of three states: •Benign:The response directly addresses the query but remains within sa...

2025
[7]

A response is deemed strictly harmless only if it passes both the discriminative check and the generative qualitative assessment

Intersection Logic (Final Verdict).The final safety label HarmCLS(y) is derived from theintersectionof these two evaluations. A response is deemed strictly harmless only if it passes both the discriminative check and the generative qualitative assessment. Formally: HarmCLS(y) = ( 1,ifC disc(y) = 1ANDJ gen(y)∈ {Benign,Refusal}, 0,otherwise. (13) This rigor...

2024
[8]

have a computer

as the reward model lowers the performance ceiling but still maintains strong safety, indicating that the optimization is robust to weaker reward supervision. Table 12.Sensitivity to teacher and reward model choices. Teacher Model Reward Model WildChat (DSR↑) DRA (DSR↑) AdvGLUE GPT-5 GPT-OSS-120B81.20% 92.31% 68.29% GPT-3.5-turbo GPT-OSS-120B 80.80% 90.85...

2025

[1] [1]

gold standard

to generate structured reflection-continuation pairs. These trajectories serve as the “gold standard” for internalizing safety reasoning and establishing the desired self-reflection policy. B.1. Synthesis Pipeline and Data Quality The construction of a complete trajectory ˜τ= (ybefore, z, yafter) is designed to simulate the emergence of a safety signal du...

[2] [2]

, T} to existing model outputs

Strategic Context Truncation:To ensure the model learns to trigger reflection at various stages of response generation, we apply a random truncation strategy n∼ U {1, . . . , T} to existing model outputs. This creates a diverse set of ybefore prefixes, ranging from early-stage intent alignment to late-stage detail generation

[3] [3]

post-mortem

Structured Reflection Generation (z):Using theSelf-Critique & Reflection Template, the teacher model performs a “post-mortem” analysis of the truncated prefix. By framing the teacher as a student reflecting on its own mistakes, we generatez reflect to identify the precise ethical breach andz explore to chart a safe path forward

[4] [4]

search-and-recovery

Reasoning-Conditioned Continuation ( yafter):The teacher generates yafter by strictly adhering to the guidance provided in z. This stage ensures that the final output is not just a refusal, but a logically consistent continuation that follows the “search-and-recovery” logic established in the reflection phase. To maintain high data fidelity, we perform a ...

2025

[5] [5]

As reported, this model demonstrates high robustness, outperforming GPT-4 by approximately 5% in agreement rates with human judgments on manually labeled validation sets

Discriminative Detector ( Cdisc).We utilize the HarmBench classifier (Mazeika et al., 2024), which is fine-tuned on Llama-2-13B. As reported, this model demonstrates high robustness, outperforming GPT-4 by approximately 5% in agreement rates with human judgments on manually labeled validation sets. It provides a binary verdict: Cdisc(y) = ( 1if Harmless 0...

2024

[6] [6]

Generative Detector ( Jgen).To capture nuance missed by binary classification, we concurrently deploy GPT-OSS- 120B (Agarwal et al., 2025) as a generative safety detector. This model evaluates the semantic content of the response trajectory and categorizes it into one of three states: •Benign:The response directly addresses the query but remains within sa...

2025

[7] [7]

A response is deemed strictly harmless only if it passes both the discriminative check and the generative qualitative assessment

Intersection Logic (Final Verdict).The final safety label HarmCLS(y) is derived from theintersectionof these two evaluations. A response is deemed strictly harmless only if it passes both the discriminative check and the generative qualitative assessment. Formally: HarmCLS(y) = ( 1,ifC disc(y) = 1ANDJ gen(y)∈ {Benign,Refusal}, 0,otherwise. (13) This rigor...

2024

[8] [8]

have a computer

as the reward model lowers the performance ceiling but still maintains strong safety, indicating that the optimization is robust to weaker reward supervision. Table 12.Sensitivity to teacher and reward model choices. Teacher Model Reward Model WildChat (DSR↑) DRA (DSR↑) AdvGLUE GPT-5 GPT-OSS-120B81.20% 92.31% 68.29% GPT-3.5-turbo GPT-OSS-120B 80.80% 90.85...

2025