pith. sign in

arxiv: 2606.22166 · v1 · pith:CEYDB3F4new · submitted 2026-06-20 · 💻 cs.DL · cs.AI· cs.LG

Rebuttals Move Peer-Review Scores, but Initial-Review Structure Bounds the Movement

Mathieu Louis , Tibo Vanleke , Vincent Ginis , Andres Algaba This is my paper

Pith reviewed 2026-06-26 10:40 UTC · model grok-4.3

classification 💻 cs.DL cs.AIcs.LG
keywords peer reviewrebuttalsscore movementLLM measurementreviewer exchangesICLRAUC predictioninitial review structure
0
0 comments X

The pith

Rebuttals move peer-review scores but initial-review structure bounds how far they shift.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper measures the effect of author rebuttals on reviewer scores at ICLR by tracking 73,000 reviewer trajectories that include externally archived pre- and post-rebuttal scores. It uses LLMs solely as instruments to recover the score that review text would have implied before rebuttal, then shows that the resulting text-score offset predicts whether scores later increase. In the subset of engaged rebuttals, features drawn from the initial review already forecast most of the observed movement, while details of the actual author-reviewer exchange add only modest extra predictive power. The work therefore claims that rebuttals can change scores yet remain constrained by how the first round of reviews was structured.

Core claim

Rebuttals can move scores, but measurable movement is bounded by initial-review structure, and robust exchange signals are mostly rebuttal failure modes. In the rebuttal-engaged benchmark, initial-review structure alone predicts score movement with AUC 0.747; adding a 44-feature taxonomy of resolved exchanges raises AUC only to 0.804. Score-increase rates rise from 8.3 percent when review text reads below the assigned score to 31.9 percent when it reads above.

What carries the argument

The text-score offset obtained by prompting an LLM to predict the pre-rebuttal score implied by score-stripped review text, which then serves as a predictor of later score movement.

If this is right

  • Score-increase rates vary from 8.3 percent to 31.9 percent according to whether review text reads below or above the assigned score.
  • Initial-review structure alone yields an AUC of 0.747 for predicting whether scores will move.
  • A 44-feature taxonomy of resolved exchanges improves that AUC to 0.804.
  • Twenty-three exchange features replicate across models and a held-out year under Bonferroni correction.
  • Strong, replicable exchange signals predominantly mark rebuttal failure rather than success.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If initial structure already forecasts most movement, then interventions aimed at clearer first-round scoring guidelines could have larger effects than changes to the rebuttal window.
  • The modest added value of exchange features suggests the rebuttal phase functions more as a clarification step than as a broad opportunity to reverse initial judgments.
  • The same LLM-based offset measurement could be applied to other conferences to test whether the bounding effect of initial structure generalizes.

Load-bearing premise

The assumption that LLM predictions of implied pre-rebuttal scores from score-stripped review text give a valid and unbiased measurement of the text-score offset.

What would settle it

In a fresh dataset containing recorded pre-rebuttal scores, the LLM-derived text-score offsets fail to separate score-increase rates into the reported 8.3 percent versus 31.9 percent bands.

Figures

Figures reproduced from arXiv: 2606.22166 by Andres Algaba, Mathieu Louis, Tibo Vanleke, Vincent Ginis.

Figure 1
Figure 1. Figure 1: ICLR rebuttals expose both structural limits and measurable content signals. a, Measurement layers and outcome flow: OpenReview records, score predictions from stripped review text, exchange-feature labels, and the external before/after score outcome. b, AUC across the three observation horizons (initial review, resolved exchange, panel discussion). c, Twenty-three of 44 exchange features replicate across … view at source ↗
Figure 2
Figure 2. Figure 2: gives the two diagnostics that make the score decoder useful: score recovery from stripped archived review text and the offset gradient linking text–score disagreement to later score movement. We first test whether score-stripped archived review text contains a recoverable numerical judgment. Before sending a review to Gemini, we remove every numeric scoring field the reviewer filled in, including soundnes… view at source ↗
Figure 3
Figure 3. Figure 3: The Bonferroni-core exchange features replicate across an independent validator and a held-out year. Each row is one of the 23 induced features that cross Bonferroni in both tests, plotted as an odds ratio for score increase versus same/decrease (log scale). Test 1: ICLR 2025 induction sample (n = 1,000, blue circles). Test 2: held-out ICLR 2024 interactions (n = 5,730, orange squares). The full 44-feature… view at source ↗
Figure 4
Figure 4. Figure 4: Only a subset of exchange features retains signal after initial-review controls. Positive-polarity features (blue) and negative-polarity features (orange) on a shared log-OR axis. Large markers: polarity-isolated conditional OR after controlling for initial-review structure. Small grey squares: univariate marginal OR. Stars mark the 14 features that pass the asymmetric screen (4 positives, 10 negatives). S… view at source ↗
Figure 5
Figure 5. Figure 5: The resolved reviewer-author exchange adds a bounded predictive gain. ROC (top) and precision-recall (bottom) curves for random forest (solid blue) and L2-regularised logistic regression (dashed orange) on the temporal split (train ICLR 2024 n = 5,710, test stratified ICLR 2025 n = 995). Columns (left to right): H0 initial review, H1 resolved exchange, H2 panel discussion. Stratum lifts in Appendix Table A… view at source ↗
read the original abstract

Author rebuttals are the main post-submission window in peer review, but their effect on reviewer scores remains hard to measure because score updates mix rebuttal content with initial score position, paper-level consensus, reviewer confidence, and discussion dynamics. We study ICLR 2024-2025 using 73,000 reviewer trajectories with externally archived pre- and post-rebuttal scores, and use LLMs only as measurement instruments. Gemini Flash 3.0 predicts implied pre-rebuttal scores from score-stripped review text. The resulting text-score offset predicts later movement, with score-increase rates rising from 8.3% when text reads below the assigned score to 31.9% when it reads above. Claude Opus 4.6 induces, and outcome-blinded Gemini Flash 3.0 validates, a 44-feature taxonomy of resolved reviewer-author exchanges, where 23 features replicate across model and held-out year under Bonferroni correction. In the rebuttal-engaged benchmark (n=6,705), initial-review structure already predicts much score movement (AUC=0.747, minimal AUC=0.696), while adding the resolved exchange raises AUC to 0.804. Rebuttals can move scores, but measurable movement is bounded by initial-review structure, and robust exchange signals are mostly rebuttal failure modes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper analyzes 73,000 reviewer trajectories from ICLR 2024-2025 with pre- and post-rebuttal scores. It uses Gemini Flash 3.0 to predict implied pre-rebuttal scores from score-stripped review text, derives a text-score offset that predicts score increases (8.3% below to 31.9% above), and induces a 44-feature taxonomy of resolved exchanges via Claude Opus 4.6 with outcome-blinded validation and cross-year replication of 23 features. In the rebuttal-engaged subset (n=6,705), initial-review structure alone yields AUC 0.747 for score movement while adding resolved exchanges raises it to 0.804. The central claim is that rebuttals move scores but measurable movement is bounded by initial-review structure, with robust exchange signals mostly indicating rebuttal failure modes.

Significance. If the LLM-based measurement instruments hold, the work supplies large-scale, replicable evidence on rebuttal dynamics that quantifies both the potential for score change and its structural limits. Strengths include the 73,000-trajectory dataset, outcome-blinded cross-model validation of the taxonomy, and held-out-year replication under Bonferroni correction. These elements allow falsifiable claims about the relative predictive power of initial structure versus post-rebuttal exchanges.

major comments (2)
  1. [Abstract / implied-score prediction] Abstract and implied-score prediction section: Gemini Flash 3.0 is used to produce the text-score offset that drives the headline movement rates (8.3% → 31.9%) and the AUC results, yet the manuscript reports no human-rated ground-truth set, calibration procedure, or inter-annotator agreement for this regression task. The only cross-model step described is for the separate 44-feature taxonomy. This measurement step is load-bearing for both the offset-movement correlation and the claim that robust signals are mostly failure modes.
  2. [Results (AUC analysis)] Results, rebuttal-engaged benchmark (n=6,705): The AUC=0.747 for initial-review structure is presented as already bounding most movement, but the manuscript provides no explicit controls or robustness checks for reviewer confidence, paper-level consensus, or discussion dynamics that the abstract itself identifies as confounders. Without these, it is unclear whether the reported bound is an artifact of unmodeled variables.
minor comments (1)
  1. [Taxonomy construction] The 44-feature taxonomy is described as capturing 'main resolved reviewer-author exchanges,' but the inducing model (Claude) and validation model (Gemini) are not compared on the same held-out set for the taxonomy itself; only replication across years is reported.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed comments. We address each major point below and indicate planned revisions to strengthen the measurement validity and robustness of the reported bounds.

read point-by-point responses

  1. Referee: [Abstract / implied-score prediction] Abstract and implied-score prediction section: Gemini Flash 3.0 is used to produce the text-score offset that drives the headline movement rates (8.3% → 31.9%) and the AUC results, yet the manuscript reports no human-rated ground-truth set, calibration procedure, or inter-annotator agreement for this regression task. The only cross-model step described is for the separate 44-feature taxonomy. This measurement step is load-bearing for both the offset-movement correlation and the claim that robust signals are mostly failure modes.

    Authors: We agree this is a substantive limitation. The implied-score regression is central to the offset-movement results and the failure-mode interpretation. The original manuscript relied on cross-year replication and downstream predictive validity rather than direct human ground truth for this task. In revision we will add a human validation study: a random sample of 300 score-stripped reviews will be independently rated by two domain experts for implied pre-rebuttal score; we will report calibration curves, mean absolute error, and inter-annotator agreement (Cohen’s κ and ICC). This directly addresses the load-bearing measurement concern while preserving the existing cross-model and replication evidence. revision: yes

  2. Referee: [Results (AUC analysis)] Results, rebuttal-engaged benchmark (n=6,705): The AUC=0.747 for initial-review structure is presented as already bounding most movement, but the manuscript provides no explicit controls or robustness checks for reviewer confidence, paper-level consensus, or discussion dynamics that the abstract itself identifies as confounders. Without these, it is unclear whether the reported bound is an artifact of unmodeled variables.

    Authors: The referee correctly notes that the abstract flags these as potential confounders yet the reported models do not include dedicated robustness checks. Initial-review structure features already encode some consensus information (e.g., score variance) and reviewer metadata, but explicit controls were omitted. In the revision we will re-estimate the AUC models with additional covariates for (i) average reviewer confidence, (ii) paper-level score dispersion, and (iii) presence of discussion threads; we will report both the change in AUC and the incremental contribution of the resolved-exchange features after these controls. This will clarify whether the structural bound remains after accounting for the identified confounders. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper extracts a text-score offset via LLM inference on score-stripped reviews and then correlates that offset with externally observed pre/post-rebuttal score changes; the central statistical claims (AUC values, movement rates) are computed from held-out trajectories and cross-model validation rather than reducing to any fitted parameter or self-defined quantity by construction. No equations, self-citations, or ansatzes are shown that would make the reported predictions equivalent to their inputs. The measurement instrument is treated as external and is outcome-blinded, satisfying the criteria for an independent derivation chain.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 1 invented entities

The central claim rests on the validity of LLM-derived implied scores and the generalizability of the induced taxonomy; no numerical free parameters are described.

axioms (2)
  • domain assumption LLMs can extract a reliable implied pre-rebuttal score from score-stripped review text that is independent of the assigned numerical score
    This is the basis for the text-score offset that predicts later movement.
  • ad hoc to paper The 44-feature taxonomy captures the main resolved reviewer-author exchanges and is not an artifact of the inducing model
    Induced by one model and validated by another with cross-year replication.
invented entities (1)
  • 44-feature taxonomy of resolved reviewer-author exchanges no independent evidence
    purpose: To categorize rebuttal content for predicting score movement
    Newly constructed for this study; no independent evidence outside the ICLR dataset is provided.

pith-pipeline@v0.9.1-grok · 5786 in / 1477 out tokens · 47302 ms · 2026-06-26T10:40:05.024690+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    ICLR 2026 statistics

    PaperCopilot. ICLR 2026 statistics. Paper Copilot statistics page, 2026

    2026

  2. [2]

    The global burden of journal peer review in the biomedical literature: Strong imbalance in the collective enterprise.PLOS ONE, 11(11):e0166387, 11 2016

    Michail Kovanis, Raphaël Porcher, Philippe Ravaud, and Ludovic Trinquart. The global burden of journal peer review in the biomedical literature: Strong imbalance in the collective enterprise.PLOS ONE, 11(11):e0166387, 11 2016

    2016

  3. [3]

    Andrew Tomkins, Min Zhang, and William D. Heavlin. Reviewer bias in single- versus double-blind peer review. Proceedings of the National Academy of Sciences, 114(48):12708–12713, 2017

    2017

  4. [4]

    On the determinants of journal rejection rates: evidence from the journal of financial economics.Financial Innovation, 12(1):52, 2026

    Karel Hrazdil, Pavel Král, Jiri Novak, and Nattavut Suwanyangyuan. On the determinants of journal rejection rates: evidence from the journal of financial economics.Financial Innovation, 12(1):52, 2026

    2026

  5. [5]

    Lawrence

    Neil D. Lawrence. The NIPS experiment. Blog post, 2014

    2014

  6. [6]

    Dauphin, Percy Liang, and Jennifer Wortman Vaughan

    Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan. Has the machine learning review process become more arbitrary as the field has grown? the NeurIPS 2021 consistency experiment, 2023

    2021

  7. [7]

    Valtchanov, Keshav R

    David Tran, Alexander V . Valtchanov, Keshav R. Ganapathy, Raymond Feng, Eric Victor Slud, Micah Goldblum, and Tom Goldstein. An open review of openreview: A critical analysis of the machine learning conference review process. ICLR 2021 submission, 2020

    2021

  8. [8]

    Alexander Goldberg, Ivan Stelmakh, Kyunghyun Cho, Alice Oh, Alekh Agarwal, Danielle Belgrave, and Ni- har B. Shah. Peer reviews of peer reviews: A randomized controlled trial and other experiments.PLOS ONE, 20(4):e0320444, 2025

    2025

  9. [9]

    Bartlett, Suyog H

    Sebastian Musslick, Laura K. Bartlett, Suyog H. Chandramouli, Marina Dubova, Fernand Gobet, Thomas L. Griffiths, Jessica Hullman, Ross D. King, J. Nathan Kutz, Christopher G. Lucas, Suhas Mahesh, Franco Pestilli, Sabina J. Sloman, and William R. Holmes. Automating the practice of science: Opportunities, challenges, and implications.Proceedings of the Nati...

    2025

  10. [10]

    Zhehui Liao, Maria Antoniak, Inyoung Cheong, Evie Yu-Yen Cheng, Ai-Heng Lee, Kyle Lo, Joseph Chee Chang, and Amy X. Zhang. LLMs as research tools: A large scale survey of researchers’ usage and perceptions, 2024

    2024

  11. [11]

    Scientific production in the era of large language models.Science, 390(6779):1240–1243, 2025

    Keigo Kusumegi, Xinyu Yang, Paul Ginsparg, Mathijs de Vaan, Toby Stuart, and Yian Yin. Scientific production in the era of large language models.Science, 390(6779):1240–1243, 2025

    2025

  12. [12]

    Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers

    Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. InInternational Conference on Learning Representations, volume 2025, pages 94003–94092, 2025

    2025

  13. [13]

    Researchagent: Iterative research idea generation over scientific literature with large language models

    Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (V olume 1: Long Papers), pa...

    2025

  14. [14]

    Agent laboratory: Using LLM agents as research assistants

    Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using LLM agents as research assistants. InFindings of the Association for Computational Linguistics: EMNLP 2025, pages 5977–6043, Suzhou, China, November 2025. Association for Computational Linguistics

    2025

  15. [15]

    Autonomous chemical research with large language models.Nature, 624(7992):570–578, 2023

    Daniil A Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical research with large language models.Nature, 624(7992):570–578, 2023

    2023

  16. [16]

    The virtual lab of ai agents designs new sars-cov-2 nanobodies.Nature, 646(8085):716–723, 2025

    Kyle Swanson, Wesley Wu, Nash L Bulaong, John E Pak, and James Zou. The virtual lab of ai agents designs new sars-cov-2 nanobodies.Nature, 646(8085):716–723, 2025

    2025

  17. [17]

    Towards end-to-end automation of AI research.Nature, 651(8107):914–919, 2026

    Chris Lu, Cong Lu, Robert Tjarko Lange, Yutaro Yamada, Shengran Hu, Jakob Foerster, David Ha, and Jeff Clune. Towards end-to-end automation of AI research.Nature, 651(8107):914–919, 2026

    2026

  18. [18]

    Why most published research findings are false.PLoS medicine, 2(8):e124, 2005

    John PA Ioannidis. Why most published research findings are false.PLoS medicine, 2(8):e124, 2005

    2005

  19. [19]

    Estimating the reproducibility of psychological science.Science, 349(6251):aac4716, 2015

    Open Science Collaboration. Estimating the reproducibility of psychological science.Science, 349(6251):aac4716, 2015

    2015

  20. [20]

    Evaluating replicability of laboratory experiments in economics.Science, 351(6280):1433–1436, 2016

    Colin F Camerer, Anna Dreber, Eskil Forsell, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Johan Almenberg, Adam Altmejd, Taizan Chan, et al. Evaluating replicability of laboratory experiments in economics.Science, 351(6280):1433–1436, 2016. 10 Rebuttals Move Peer-Review Scores

    2016

  21. [21]

    Science of science.Science, 359(6379):eaao0185, 2018

    Santo Fortunato, Carl T Bergstrom, Katy Börner, James A Evans, Dirk Helbing, Staša Milojevi´c, Alexander M Petersen, Filippo Radicchi, Roberta Sinatra, Brian Uzzi, et al. Science of science.Science, 359(6379):eaao0185, 2018

    2018

  22. [22]

    A dataset of peer reviews (PeerRead): Collection, insights and NLP applications

    Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. In Marilyn Walker, Heng Ji, and Amanda Stent, editors,Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Li...

    2018

  23. [23]

    Argument mining for understanding peer reviews

    Xinyu Hua, Mitko Nikolov, Nikhil Badugu, and Lu Wang. Argument mining for understanding peer reviews. In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V olume 1 (Long and Short Papers), pages 2131–2137...

    2019

  24. [24]

    NLPeer: A unified resource for the computational study of peer review

    Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. NLPeer: A unified resource for the computational study of peer review. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors,Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 5049–5073, Toronto, Canada, July 2023. Association for...

    2023

  25. [25]

    Kummerfeld, Anne Lauscher, Kevin Leyton-Brown, Sheng Lu, Mausam, Margot Mieskes, Aurélie Névéol, Danish Pruthi, Lizhen Qu, Roy Schwartz, Noah A

    Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen, Nils Dycke, Alexander Goldberg, Tom Hope, Dirk Hovy, Jonathan K. Kummerfeld, Anne Lauscher, Kevin Leyton-Brown, Sheng Lu, Mausam, Margot Mieskes, Aurélie Névéol, Danish Pruthi, Lizhen Qu, Roy Schwartz, Noah A. Smith, Thamar Solorio, Jingyan Wang, Xiaodan Zhu, Anna Rogers, Nihar B. Shah, and Iryna Gurevy...

    2024

  26. [26]

    DISAPERE: A dataset for discourse structure in peer review discussions

    Neha Nayak Kennard, Tim O’Gorman, Rajarshi Das, Akshay Sharma, Chhandak Bagchi, Matthew Clinton, Pranay Kumar Yelugam, Hamed Zamani, and Andrew McCallum. DISAPERE: A dataset for discourse structure in peer review discussions. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors,Proceedings of the 2022 Conference of the Nort...

    2022

  27. [27]

    Re2: A consistency-ensured dataset for full-stage peer review and multi-turn rebuttal discussions, 2025

    Daoze Zhang, Zhijian Bao, Sihang Du, Zhiyi Zhao, Kuangling Zhang, Dezheng Bao, and Yang Yang. Re2: A consistency-ensured dataset for full-stage peer review and multi-turn rebuttal discussions, 2025

    2025

  28. [28]

    Does my rebuttal matter? insights from a major NLP conference

    Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych, and Yusuke Miyao. Does my rebuttal matter? insights from a major NLP conference. In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V olume 1 (Long...

    2019

  29. [29]

    Association for Computational Linguistics

  30. [30]

    What makes a successful rebuttal in computer science conferences?: A perspective on social interaction.Journal of Informetrics, 17(3):101427, 2023

    Junjie Huang, Win-bin Huang, Yi Bu, Qi Cao, Huawei Shen, and Xueqi Cheng. What makes a successful rebuttal in computer science conferences?: A perspective on social interaction.Journal of Informetrics, 17(3):101427, 2023

    2023

  31. [31]

    What drives paper acceptance? a process-centric analysis of modern peer review, 2025

    Sangkeun Jung, Goun Pyeon, Inbum Heo, and Hyungjin Ahn. What drives paper acceptance? a process-centric analysis of modern peer review, 2025

    2025

  32. [32]

    Insights from the iclr peer review and rebuttal process, 2025

    Amir Hossein Kargaran, Nafiseh Nikeghbal, Jing Yang, and Nedjma Ousidhoum. Insights from the iclr peer review and rebuttal process, 2025

    2025

  33. [33]

    How effective is your rebuttal? identifying causal models from the OpenReview system

    Loka Li, Ibrahim Aldarmaki, Minghao Fu, Wong Yu Kang, Yunlong Deng, Qiang Huang, Jing Yang, Jin Tian, Guangyi Chen, and Kun Zhang. How effective is your rebuttal? identifying causal models from the OpenReview system. InNeurIPS 2025 Workshop CauScien: Uncovering Causality in Science, 2025

    2025

  34. [34]

    Paper copilot: Tracking the evolution of peer review in AI conferences

    Jing Yang, Qiyao Wei, and Jiaxin Pei. Paper copilot: Tracking the evolution of peer review in AI conferences. ICLR 2026 poster, 2026

    2026

  35. [35]

    Decoupling scores and text: The politeness principle in peer review, 2026

    Yingxuan Wen. Decoupling scores and text: The politeness principle in peer review, 2026

    2026

  36. [36]

    Is your paper being reviewed by an LLM? benchmarking AI text detection in peer review

    Sungduk Yu, Man Luo, Avinash Madasu, Vasudev Lal, and Phillip Howard. Is your paper being reviewed by an LLM? benchmarking AI text detection in peer review. ICLR 2026 poster, 2026

    2026

  37. [37]

    Can large language models provide useful feedback on research papers? a large-scale empirical analysis.NEJM AI, 1(8):AIoa2400196, 2024

    Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Ding, Xinyu Yang, Kailas V odrahalli, Siyu He, Daniel Smith, Yian Yin, Daniel McFarland, and James Zou. Can large language models provide useful feedback on research papers? a large-scale empirical analysis.NEJM AI, 1(8):AIoa2400196, 2024. 11 Rebuttals Move Peer-Review Scores

    2024

  38. [38]

    A large-scale randomized study of large language model feedback in peer review.Nature Machine Intelligence, 8(3):326–336, 2026

    Nitya Thakkar, Mert Yuksekgonul, Jake Silberg, Animesh Garg, Nanyun Peng, Fei Sha, Rose Yu, Carl V ondrick, and James Zou. A large-scale randomized study of large language model feedback in peer review.Nature Machine Intelligence, 8(3):326–336, 2026

    2026

  39. [39]

    The ai review lottery: Widespread ai-assisted peer reviews boost paper scores and acceptance rates.Proc

    Giuseppe Russo, Manoel Horta Ribeiro, Tim Ruben Davidson, Veniamin Veselovsky, and Robert West. The ai review lottery: Widespread ai-assisted peer reviews boost paper scores and acceptance rates.Proc. ACM Hum.-Comput. Interact., 9(7):1–28, October 2025

    2025

  40. [40]

    Wagstaff, Matthew E

    Joydeep Biswas, Sheila Schoepp, Gautham Vasan, Anthony Opipari, Arthur Zhang, Zichao Hu, Sebastian Joseph, Matthew Lease, Junyi Jessy Li, Peter Stone, Kiri L. Wagstaff, Matthew E. Taylor, and Odest Chadwicke Jenkins. AI-assisted peer review at scale: The AAAI-26 AI review pilot, 2026

    2026

  41. [41]

    Scholarpeer: A context-aware multi-agent framework for automated peer review, 2026

    Palash Goyal, Mihir Parmar, Yiwen Song, Hamid Palangi, Tomas Pfister, and Jinsung Yoon. Scholarpeer: A context-aware multi-agent framework for automated peer review, 2026

    2026

  42. [42]

    DRPG (decompose, retrieve, plan, generate): An agentic framework for academic rebuttal, 2026

    Peixuan Han, Yingjie Yu, Jingjun Xu, and Jiaxuan You. DRPG (decompose, retrieve, plan, generate): An agentic framework for academic rebuttal, 2026

    2026

  43. [43]

    Defend: Automated rebuttals for peer review with minimal author guidance, 2026

    Jyotsana Khatri and Manasi Patwardhan. Defend: Automated rebuttals for peer review with minimal author guidance, 2026

    2026

  44. [44]

    Stop automating peer review without rigorous evaluation

    Joachim Baumann, Jiaxin Pei, Sanmi Koyejo, and Dirk Hovy. Stop automating peer review without rigorous evaluation. InPost-AGI Science and Society Workshop, 2026

    2026

  45. [45]

    Yu, and Wenpeng Yin

    Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Ranran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, Chen Xing, Cheng Jiayang, Zhaowei Wang, Ying Su, Raj Sanjay Shah, Ruohao Guo, Jing Gu, Haoran Li, K...

    2024

  46. [46]

    On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists

    Seungone Kim, Dongkeun Yoon, Kiril Gashteovski, Juyoung Suk, Jinheon Baek, Pranjal Aggarwal, Ian Wu, Viktor Zaverkin, Spase Petkoski, Daniel R Schrider, et al. On the limits and opportunities of ai reviewers: Reviewing the reviews of nature-family papers with 45 expert scientists.arXiv preprint arXiv:2605.20668, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  47. [47]

    Datasheets for datasets.Communications of the ACM, 64(12):86–92, 2021

    Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets.Communications of the ACM, 64(12):86–92, 2021

    2021

  48. [48]

    Data cards: Purposeful and transparent dataset documentation for responsible AI

    Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. Data cards: Purposeful and transparent dataset documentation for responsible AI. InProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1776–1826, 2022

    2022

  49. [49]

    Croissant: A metadata format for ML-ready datasets

    Mubashara Akhtar, Omar Benjelloun, Costanza Conforti, Pieter Gijsbers, Joan Giner-Miguelez, Nitisha Jain, Michael Kuchnik, Quentin Lhoest, Pierre Marcenac, Manil Maskey, Peter Mattson, Luis Oala, Pierre Ruyssen, Rajat Shinde, Elena Simperl, Goeffry Thomas, Slava Tykhonov, Joaquin Vanschoren, Jos van der Velde, Steffen V ogler, and Carole-Jean Wu. Croissan...

    2024

  50. [50]

    Terms of use

    OpenReview. Terms of use. https://openreview.net/legal/terms, 2024. Comment licence CC-BY-4.0; metadata licence CC0; article licence per submission

    2024

  51. [51]

    ICLR 2026 response to security incident

    ICLR 2026 Program Chairs. ICLR 2026 response to security incident. ICLR Blog, 2025

    2026

  52. [52]

    A retrospective on the ICLR 2026 review process

    ICLR 2026 Program Chairs. A retrospective on the ICLR 2026 review process. ICLR Blog, 2026

    2026

  53. [53]

    A simple sequentially rejective multiple test procedure.Scandinavian Journal of Statistics, 6(2):65–70, 1979

    Sture Holm. A simple sequentially rejective multiple test procedure.Scandinavian Journal of Statistics, 6(2):65–70, 1979

    1979

  54. [54]

    With scores

    Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: A practical and powerful approach to multiple testing.Journal of the Royal Statistical Society: Series B (Methodological), 57(1):289–300, 1995. 12 Rebuttals Move Peer-Review Scores A Data and Code Availability The analysis code is available on GitHub at https://github.com/Integrated-...

    1995

  55. [55]

    the H0 feature set contains none of the column names from the H1- or H2-only buckets

  56. [56]

    the H0-minimal matrix contains only the six provenance-minimal features listed in Appendix Table A6

  57. [57]

    synthetic perturbation: changing any ext_rating_after value leaves paper_initial_mean unchanged, ruling out accidental use of post-rebuttal scores in the paper-level summary

  58. [58]

    balanced

    no individual feature has|r| ≥0.99with the binary outcome at any horizon (outcome-leak tripwire). All four tests pass at the time of writing. Models.Logistic regression: ℓ2-regularised, class_weight="balanced", max_iter= 2000 , features standard- ised at fit time. Random forest: 500 trees, min_samples_leaf= 20 , class_weight="balanced_subsample", n_jobs=−...

    2000