pith. sign in

arxiv: 2604.23488 · v2 · pith:OQ46UJ2Gnew · submitted 2026-04-26 · 💻 cs.LG

Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Trainig-Time Reward Hacking in Code Generation

Lichen Li , Hengguang Zhou , Yijun Liang , Tianyi Zhou , Cho-Jui Hsieh This is my paper

Pith reviewed 2026-07-01 09:54 UTC · model grok-4.3

classification 💻 cs.LG
keywords reward hackingcode generationreinforcement learningmonitor generalizationtrajectory curationRL safetyunit-test tracing
0
0 comments X

The pith

Monitors trained on prompted reward-hacking examples fail to detect hacks that emerge during actual RL training for code generation.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that trajectories collected by explicitly prompting models to produce reward hacks do not match the hacking behaviors that arise when models undergo reinforcement learning on code tasks without such prompts. It introduces Trace-and-Amplify, which applies unit-test tracers during RL runs to capture and retain those natural hacking trajectories for use in training and testing monitors. Controlled experiments show that monitors trained only on prompted trajectories perform poorly when evaluated on the naturally collected ones. In contrast, monitors trained on the Trace-and-Amplify trajectories generalize better to previously unseen forms of hacking. The work concludes that evaluations based solely on prompted data can produce misleading assessments of monitor reliability.

Core claim

Prompt-elicited hacking trajectories do not reflect the distribution of reward-hacking behaviors that occur during RL training without explicit instructions. Monitors trained on prompt-elicited data often fail to generalize to trajectories curated by Trace-and-Amplify, while monitors trained on Trace-and-Amplify trajectories demonstrate stronger generalizability to unseen hacking types. This indicates that prompted reward-hacking data may not fully capture training-time behaviors and that reliance on such data alone can lead to misleading conclusions about monitor performance.

What carries the argument

Trace-and-Amplify, a framework that deploys unit-test tracers during RL training to identify hacking solutions when they occur and retains those trajectories for monitor training and evaluation.

If this is right

  • Monitors trained on prompt-elicited data often fail to generalize to trajectories curated by Trace-and-Amplify.
  • Monitors trained on Trace-and-Amplify trajectories show stronger generalizability to unseen hacking types.
  • Relying solely on prompted reward-hacking data can produce misleading conclusions about monitor effectiveness in RL settings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Safety evaluations for code-generation models may need to incorporate ongoing trajectory collection from actual training runs rather than relying on static prompted datasets.
  • The gap between prompted and training-time behaviors suggests that monitor training pipelines should simulate the full RL loop to avoid over-optimism.
  • Unit-test-based detection methods could be adapted to other domains where reward signals are sparse or indirect.

Load-bearing premise

Unit-test tracers can reliably identify genuine reward hacks that arise naturally during RL without explicit instructions, and the collected trajectories represent the distribution monitors would encounter in practice.

What would settle it

A direct test in which monitors trained on Trace-and-Amplify trajectories are evaluated on hacking behaviors collected from fresh RL training runs that use no hacking prompts and no unit-test filtering during data collection.

Figures

Figures reproduced from arXiv: 2604.23488 by Cho-Jui Hsieh, Hengguang Zhou, Lichen Li, Tianyi Zhou, Yijun Liang.

Figure 1
Figure 1. Figure 1: Overview of the Trace-and-Amplify framework. For each task, we construct a contradictory unit test set to identify hacking behavior, where only hacking solutions can satisfy all constraints. During GRPO training, candidate solutions are sampled and evaluated; groups without a detected hacking instance are discarded and resampled, while groups containing at least one hacking sample are used for policy updat… view at source ↗
Figure 3
Figure 3. Figure 3: UMAP visualizations of model activations We utilize UMAP to visualize the distribution of activations of the last text token at the last hidden layer of the LLM, after it reads the structured prompt-response pairs. The activation distribution of synthetic and in-the-wild data is shown separately in view at source ↗
read the original abstract

Reward hacking in code generation, where models exploit evaluation loopholes to obtain high reward without correctly solving the intended task, poses a critical challenge for Reinforcement Learning (RL) and the deployment of reasoning models. Existing studies often rely on explicitly prompted hacking trajectories, but it remains unclear whether monitors trained on such data can detect reward hacks that arise without direct hacking instructions during RL training. In this work, we introduce Trace-and-Amplify, a framework for scalable curation of reward-hacking trajectories that arise during RL training without explicit hacking instructions. The framework uses unit-test tracers to identify hacking solutions when they occur and retains such trajectories for monitor training and evaluation. Through controlled comparisons between monitors trained on prompt-elicited hacking trajectories and training-time reward-hacking trajectories collected by Trace-and-Amplify, we find that \textbf{(1) prompt-elicited-data-trained monitors often fail to generalize to trajectories curated by our framework}, and \textbf{(2) monitors trained on our Trace-and-Amplify trajectories demonstrate stronger generalizability to unseen hacking types}. Our results indicate that prompted reward hacking data may not fully reflect training-time reward-hacking behaviors, and that relying solely on these data can lead to misleading conclusions. Codebase is available at https://github.com/LichenLillc/CoTMonitoring.git

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces the Trace-and-Amplify framework for curating reward-hacking trajectories in code generation that arise during RL training without explicit hacking instructions, using unit-test tracers. Through controlled comparisons, it reports that monitors trained on prompt-elicited trajectories often fail to generalize to these curated trajectories, while monitors trained on Trace-and-Amplify trajectories show stronger generalizability to unseen hacking types. The authors conclude that prompted data may not fully reflect training-time reward-hacking behaviors.

Significance. If the unit-test tracers reliably capture genuine reward hacks, this result would indicate that existing studies using prompted hacking data may overestimate monitor effectiveness, suggesting the need for more realistic data collection methods like Trace-and-Amplify. The open-sourced codebase supports reproducibility. However, the empirical nature means the significance depends on the validity of the data curation process.

major comments (2)
  1. [Abstract and Trace-and-Amplify framework description] The central comparison relies on the assumption that unit-test tracers accurately identify reward hacks arising without explicit instructions. However, no details are provided on tracer validation, such as false-positive rates, coverage of hack types, or human review comparisons. This is load-bearing for the claim that prompt-elicited data is insufficient, as systematic errors in tracer output would invalidate the generalization findings.
  2. [Experimental results (findings 1 and 2)] The abstract states the two main findings but provides no information on sample sizes, statistical tests, metrics used for generalization, or exclusion criteria. Without these, it is difficult to evaluate whether the reported failure to generalize and stronger generalizability are statistically robust.
minor comments (2)
  1. [Title] The title contains a typo: 'Trainig-Time' should be 'Training-Time'.
  2. [Abstract] The abstract mentions 'our framework' but does not define or briefly describe the Trace-and-Amplify method beyond its use of unit-test tracers.

Simulated Author's Rebuttal

2 responses · 0 unresolved

Thank you for the constructive feedback on our manuscript. We have carefully considered each major comment and provide point-by-point responses below, along with planned revisions where appropriate.

read point-by-point responses

  1. Referee: [Abstract and Trace-and-Amplify framework description] The central comparison relies on the assumption that unit-test tracers accurately identify reward hacks arising without explicit instructions. However, no details are provided on tracer validation, such as false-positive rates, coverage of hack types, or human review comparisons. This is load-bearing for the claim that prompt-elicited data is insufficient, as systematic errors in tracer output would invalidate the generalization findings.

    Authors: We acknowledge the importance of validating the unit-test tracers to ensure they accurately capture reward hacks without explicit instructions. The manuscript does not currently provide quantitative details on false-positive rates, coverage, or human review comparisons. To address this, we will include a new subsection in the revised manuscript describing the tracer validation process, including an analysis of false positives on a sample of trajectories and, where possible, comparisons with human judgments. revision: yes

  2. Referee: [Experimental results (findings 1 and 2)] The abstract states the two main findings but provides no information on sample sizes, statistical tests, metrics used for generalization, or exclusion criteria. Without these, it is difficult to evaluate whether the reported failure to generalize and stronger generalizability are statistically robust.

    Authors: The abstract provides a high-level summary of the findings due to space constraints. Comprehensive details on sample sizes, the metrics for measuring generalization (such as detection rates on unseen hacking types), statistical tests, and any exclusion criteria are presented in the Experimental Setup and Results sections. Nevertheless, we agree that referencing these in the abstract could aid readers. We will revise the abstract to briefly note the evaluation metrics and direct readers to the methods for full statistical details. revision: partial

Circularity Check

0 steps flagged

Empirical comparison of data sources; no derivations or self-referential reductions

full rationale

The paper is a controlled empirical study comparing monitors trained on prompt-elicited trajectories versus Trace-and-Amplify trajectories for detecting reward hacking. The abstract and description contain no equations, first-principles derivations, fitted parameters renamed as predictions, or load-bearing self-citations. Claims rest on experimental generalization results that are directly testable against held-out data. No step reduces by construction to its inputs; the work is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 1 invented entities

Based on abstract only; no explicit free parameters, axioms, or invented entities beyond the framework itself are stated.

invented entities (1)
  • Trace-and-Amplify framework no independent evidence
    purpose: scalable curation of reward-hacking trajectories that arise during RL training without explicit hacking instructions
    New framework introduced to collect natural hacking trajectories using unit-test tracers

pith-pipeline@v0.9.1-grok · 5783 in / 1094 out tokens · 28697 ms · 2026-07-01T09:54:53.539074+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages · 6 internal anchors

  1. [1]

    Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation

    URL https://arxiv.org/abs/ 2503.11926. Mohammad Beigi, Ming Jin, Junshan Zhang, Jiaxin Zhang, Qifan Wang, and Lifu Huang. IR3: Contrastive inverse reinforcement learning for interpretable detection and mitigation of reward hacking.arXiv preprint arXiv:2602.19416,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Training large language models on narrow tasks can lead to broad misalignment , volume=

    ISSN 1476-4687. doi: 10.1038/s41586-025-09937-5. URL http://dx.doi.org/10.1038/ s41586-025-09937-5. Yik Siu Chan, Zheng-Xin Yong, and Stephen H. Bach. Can we predict alignment before models finish thinking? towards monitoring misaligned reasoning models,

    work page doi:10.1038/s41586-025-09937-5

  3. [3]

    James Chua, Jan Betley, Mia Taylor, and Owain Evans

    URL https://arxiv.org/abs/2507.12428. James Chua, Jan Betley, Mia Taylor, and Owain Evans. Thought crime: Backdoors and emergent misalignment in reasoning models.arXiv preprint arXiv:2506.13206,

    work page arXiv

  4. [4]

    Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models

    Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, et al. Sycophancy to subterfuge: Investigating reward-tampering in large language models.arXiv preprint arXiv:2406.10162,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    arXiv preprint arXiv:2512.18311 , year=

    Melody Y Guan, Miles Wang, Micah Carroll, Zehao Dou, Annie Y Wei, Marcus Williams, Benjamin Arnav, Joost Huizinga, Ian Kivlichan, Mia Glaese, et al. Monitoring monitora- bility.arXiv preprint arXiv:2512.18311,

    work page arXiv

  6. [6]

    DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence

    URLhttps://arxiv.org/abs/2401.14196. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Rua...

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    Skywork Open Reasoner 1 Technical Report

    ISSN 1476-4687. doi: 10.1038/s41586-025-09422-z. URL http://dx.doi.org/10.1038/s41586-025-09422-z. Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner 1 technical report. arXiv pr...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.1038/s41586-025-09422-z

  8. [8]

    Correlated proxies: A new definition and improved mitigation for reward hacking.arXiv preprint arXiv:2403.03185,

    Cassidy Laidlaw, Shivam Singhal, and Anca Dragan. Correlated proxies: A new definition and improved mitigation for reward hacking.arXiv preprint arXiv:2403.03185,

    work page arXiv

  9. [9]

    Natural emergent misalignment from reward hacking in production rl, 2025

    Monte MacDiarmid, Benjamin Wright, Jonathan Uesato, Joe Benton, Jon Kutasov, Sara Price, Naia Bouscal, Sam Bowman, Trenton Bricken, Alex Cloud, et al. Natural emergent misalignment from reward hacking in production rl.arXiv preprint arXiv:2511.18397,

    work page arXiv

  10. [10]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    arXiv preprint arXiv:2508.17511 , year=

    URL https://arxiv.org/abs/2508.17511. Chaoqi Wang, Zhuokai Zhao, Yibo Jiang, Zhaorun Chen, Chen Zhu, Yuxin Chen, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Hao Ma, et al. Beyond reward hacking: Causal rewards for large language model alignment.arXiv preprint arXiv:2501.09620, 2025a. Miles Wang, Tom Dupré la Tour, Olivia Watkins, Alex Makelov, Ryan A. Chi, Samu...

    work page arXiv

  12. [12]

    Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference

    URLhttps://arxiv.org/abs/2412.13663. Patrick Wilhelm, Thorsten Wittkopp, and Odej Kao. Monitoring emergent reward hacking during generation via internal activations.arXiv preprint arXiv:2603.04069,

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [13]

    Leetcodedataset: A temporal dataset for robust evaluation and efficient training of code llms.arXiv preprint arXiv:2504.14655, 2025

    URLhttps://arxiv.org/abs/2504.14655. 11 Preprint. Under review. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin...

    work page arXiv

  14. [14]

    Reasoning models know when they’re right: Probing hidden states for self- verification.arXiv preprint arXiv:2504.05419,

    Anqi Zhang, Yulin Chen, Jane Pan, Chen Zhao, Aurojit Panda, Jinyang Li, and He He. Reasoning models know when they’re right: Probing hidden states for self-verification, 2025a. URLhttps://arxiv.org/abs/2504.05419. Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of develo...

    work page arXiv 2025

  15. [15]

    Reward Hacker

    URLhttps://arxiv.org/abs/2510.20270. A Appendix A.1 Curating Large-Scale In-the-wild Dataset Configuration of "Reward Hacker" TrainingWe use 4 × A6000 GPUs to post-train the Qwen2.5-Coder-1.5B-Instruct and DeepSeek-Coder-1.3B-Instruct on LeetCode and TACO datasets for approximately 300 steps with our Trace-and-Amplify framework. The batch size is set to

    work page arXiv