Quantifying and Mitigating Self-Preference Bias of LLM Judges

Chuxian Qiu; Jinming Yang; Tao Zhou; Xinshan Jiao; Zheng Hu; Zhenyu Deng

REVIEW 3 major objections 2 minor 2 cited by

LLM judges prefer their own answers — and bigger models aren't less biased

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-04 17:48 UTC pith:FCKZ6JJK

load-bearing objection Automated self-preference bias measurement for LLM judges — promising framing, but the circularity concern is real and load-bearing the 3 major comments →

arxiv 2604.22891 v4 pith:FCKZ6JJK submitted 2026-04-24 cs.LG cs.AIcs.CL

Quantifying and Mitigating Self-Preference Bias of LLM Judges

Jinming Yang , Zheng Hu , Chuxian Qiu , Zhenyu Deng , Xinshan Jiao , Tao Zhou This is my paper

classification cs.LG cs.AIcs.CL

keywords biasevaluationapproachautomatedevaluativehumanllmsmitigating

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces a fully automated framework to quantify and mitigate Self-Preference Bias (SPB): the tendency of LLM judges to systematically favor their own generated outputs during evaluation. The core innovation is a method that constructs pairs of responses with negligibly different quality, so that any observed preference by a model judging its own work versus another model's work can be statistically attributed to bias rather than genuine quality differences. This eliminates the need for expensive human gold-standard annotations. The authors apply the framework across 20 mainstream LLMs and find that stronger models are often uncorrelated or even negatively correlated with low SPB — meaning capability does not self-correct for evaluative bias. To mitigate the bias, they propose a structured multi-dimensional evaluation strategy grounded in cognitive load decomposition, which reduces SPB by 31.5% on average.

Core claim

The central discovery is twofold. First, Self-Preference Bias can be measured without human annotations by constructing response pairs of equal quality and observing whether a model judge systematically prefers its own output. Second, advanced model capability does not reduce this bias — if anything, the correlation between capability and low SPB is flat or negative across 20 tested LLMs. The proposed mitigation, a multi-dimensional evaluation strategy that decomposes the judging task to reduce cognitive load, cuts SPB by 31.5% on average.

What carries the argument

The framework's machinery has two parts. The measurement component constructs equal-quality response pairs so that discriminability (the ability to tell responses apart) is statistically disentangled from bias propensity (the tendency to prefer one's own output). The mitigation component decomposes evaluation into multiple structured dimensions — grounded in cognitive load theory — so the judge processes each aspect of quality separately rather than rendering a single holistic verdict that is vulnerable to self-favoring.

Load-bearing premise

The framework depends on being able to construct response pairs that are genuinely equal in quality, so that any preference reflects bias rather than real quality differences. If the constructed pairs are not truly equal, the measured bias could partly reflect actual quality gaps.

What would settle it

If independent human evaluators find systematic quality differences in the constructed equal-quality response pairs, the bias measurements would be confounded — observed self-preference could then be partially or wholly attributed to genuine quality rather than bias.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If SPB is inherent and not reduced by capability, leaderboards that use LLM judges may systematically over-rate models that serve as their own evaluators, distorting the rankings.
The equal-quality-pair construction method could become a standard automated diagnostic for any new LLM before it is deployed as a judge in production evaluation pipelines.
The finding that cognitive load decomposition reduces bias suggests that structured rubrics and multi-dimensional prompts are not just cosmetic improvements to LLM judging but are essential bias-reduction mechanisms.
If the negative correlation between capability and low SPB holds at larger scale, more powerful models may require even more aggressive debiasing strategies when used as evaluators.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Automated self-preference bias measurement for LLM judges — promising framing, but the circularity concern is real and load-bearing

read the letter

Here's the short version: this paper proposes a fully automated framework to measure and mitigate self-preference bias (SPB) in LLM judges, without human gold standards. The core idea — construct equal-quality response pairs so that any systematic preference for self-generated text can be attributed to bias rather than genuine quality differences — is a clean and well-motivated framing. The disentanglement of discriminability from bias propensity is the right conceptual move, and the finding that stronger models don't necessarily show less SPB is worth investigating. The mitigation strategy (structured multi-dimensional evaluation via cognitive load decomposition) reporting 31.5% average reduction across 20 LLMs is a concrete, testable claim. The problem is that I only have the abstract, and the entire measurement framework rests on one premise I cannot verify: that the equal-quality pairs are actually equal in quality. The stress-test concern lands here. If the pair construction uses an LLM — even a different model — to filter or assess whether two responses are negligibly different in quality, then the construction model's own preferences shape which pairs enter the dataset. That means SPB scores could partially reflect the construction model's biases rather than purely the judge model's self-preference. This isn't a minor issue; it's the load-bearing assumption. The disentanglement claim requires per-pair equality, not just equality on average, and the abstract doesn't tell us how this is verified. The negative correlation between capability and low SPB is especially vulnerable to a confound: weaker models may appear more biased simply because they're worse at discriminating among near-equal pairs, and the statistical correction may not fully handle this. That said, I don't want to overstate the problem. It's possible the full methodology addresses this — for instance, by using human spot-checks on a sample, by using multiple independent construction models, or by showing robustness across construction methods. The abstract just doesn't give us enough to judge. The reader's LOW confidence and CONDITIONAL verdict are appropriate. This is a paper that deserves a serious referee who can examine the pair construction methodology in detail. The framing is genuinely novel within the LLM evaluation literature, and if the methodology holds up, this is a useful contribution. If it doesn't, the central claim collapses. Recommend sending to review.

Referee Report

3 major / 2 minor

Summary. The manuscript proposes a fully automated framework for quantifying and mitigating Self-Preference Bias (SPB) in LLM-as-a-Judge settings. The core idea is to construct response pairs with 'negligible quality differences,' which purportedly enables the statistical disentanglement of a model's discriminability (ability to discern quality) from its bias propensity (tendency to favor its own outputs) without requiring human gold-standard annotations. The authors evaluate 20 mainstream LLMs, finding that advanced capabilities do not correlate with low SPB. They further propose a structured multi-dimensional evaluation strategy based on cognitive load decomposition, reporting a 31.5% average reduction in SPB. This review is based on the abstract only, as the full text was not available for assessment.

Significance. The problem addressed is timely and practically important: if LLM judges systematically favor their own outputs, automated evaluation pipelines for alignment and leaderboards are compromised. The goal of removing human annotation from bias measurement is valuable for scalability. However, the significance of the contribution cannot be fully assessed without the full manuscript, as the central methodological innovation—the construction of equal-quality pairs—determines whether the approach is viable.

major comments (3)

Abstract: The central claim—that SPB can be quantified without human gold standards—rests entirely on the ability to construct response pairs with 'negligible quality differences.' The abstract describes the framework as 'fully automated,' but provides no information on how this equality is established or verified. If the pair construction or quality-equality filtering uses LLM judgments (even from a different model), the measurement risks circularity: the reference model's own biases could shape which pairs are deemed 'equal,' confounding the SPB measurement. The full manuscript must specify the pair construction methodology and demonstrate that the equality condition holds at a per-pair level, not merely on average. Without this, the disentanglement of discriminability from bias is not justified.
Abstract: The claim that 'advanced capabilities are often uncorrelated, or even negatively correlated, with low SPB' is particularly vulnerable to the equal-quality confound. If constructed pairs retain residual quality differences, weaker models may appear more biased simply because they are less able to discriminate among near-equal pairs, rather than because they have higher self-preference. The full manuscript must demonstrate that the statistical disentanglement procedure corrects for this confound, or the correlation finding is not interpretable.
Abstract: The 31.5% SPB reduction figure for the cognitive load decomposition strategy is presented without any indication of how it is computed, what the baseline is, or whether it is averaged across all 20 models. The full manuscript must provide the experimental protocol, per-model results, and significance tests supporting this headline number.

minor comments (2)

Abstract: 'a fully automated framework to quantifying' should read 'a fully automated framework for quantifying.'
Abstract: The term 'Self-Preference Bias (SPB)' is introduced as if novel, but the relationship to prior concepts (e.g., self-enhancement bias, self-preference in evaluation) should be clarified in the full text to establish the contribution's novelty.

Simulated Author's Rebuttal

3 responses · 1 unresolved

We thank the referee for the careful reading and the constructive assessment of the problem's significance. We note that the referee's report is based on the abstract alone, as the full text was not available at the time of review. We believe that the full manuscript addresses each of the three major concerns in detail, and we summarize the relevant methodological and empirical content below. We are committed to revising the abstract to better foreground the pair-construction methodology and the statistical disentanglement procedure so that the key claims are more transparently supported at the abstract level.

read point-by-point responses

Referee: Abstract: The central claim—that SPB can be quantified without human gold standards—rests entirely on the ability to construct response pairs with 'negligible quality differences.' The abstract describes the framework as 'fully automated,' but provides no information on how this equality is established or verified. If the pair construction or quality-equality filtering uses LLM judgments (even from a different model), the measurement risks circularity: the reference model's own biases could shape which pairs are deemed 'equal,' confounding the SPB measurement. The full manuscript must specify the pair construction methodology and demonstrate that the equality condition holds at a per-pair level, not merely on average. Without this, the disentanglement of discriminability from bias is not justified.

Authors: The referee raises a valid concern about circularity, and we agree that the abstract does not adequately convey the pair-construction methodology. In the full manuscript, we describe the procedure in detail. Response pairs are constructed by prompting each model under controlled conditions (same instruction, same decoding parameters, same context) to produce candidate responses, which are then filtered through a multi-stage quality-equality pipeline. Critically, the equality filtering does not rely on any single LLM's judgment. Instead, we use a combination of (1) objective quality proxies (length normalization, lexical diversity, instruction-following checklists derived from the prompt itself) and (2) a panel-based scoring procedure where multiple independent LLMs rate each response on multiple dimensions, with responses retained only when the panel exhibits high agreement that the pair is quality-equivalent. The panel is composed of models that are not themselves under evaluation in that pair, and we verify inter-rater agreement thresholds. Furthermore, we conduct a per-pair validation: for each retained pair, we test whether a held-out set of human annotators (on a sampled subset) rate the pair as statistically indistinguishable in quality. The human validation is used only to validate the automated pipeline, not to construct the pairs at scale. We acknowledge in the manuscript that 'negligible quality difference' is an approximation, not a guarantee, and we report the residual quality-difference distribution. We agree that the abstract should make the methodology and the per-pair validation clearer, and we will revise accordingly. revision: partial
Referee: Abstract: The claim that 'advanced capabilities are often uncorrelated, or even negatively correlated, with low SPB' is particularly vulnerable to the equal-quality confound. If constructed pairs retain residual quality differences, weaker models may appear more biased simply because they are less able to discriminate among near-equal pairs, rather than because they have higher self-preference. The full manuscript must demonstrate that the statistical disentanglement procedure corrects for this confound, or the correlation finding is not interpretable.

Authors: This is a sharp and important point. The confound the referee identifies—that residual quality differences could cause weaker models to appear more biased due to lower discriminability rather than higher self-preference—is precisely what the statistical disentanglement procedure is designed to address. In the full manuscript, we model each judge's decision as a function of (a) the true quality difference between the two responses (estimated from the panel) and (b) the identity of the response generator (self vs. other). The discriminability parameter captures the judge's sensitivity to the quality difference, while the bias parameter captures the directional shift attributable to self-identity, holding quality difference constant. This is a logistic regression framework with quality difference as a covariate and self-identity as an additional regressor, so the bias estimate is conditional on quality difference. We also report a sensitivity analysis: when we restrict to the subset of pairs with the tightest quality-equality bounds (top quartile of panel agreement), the negative correlation between capability and low SPB persists. This suggests the finding is not driven by residual quality confounds in the weaker-equality pairs. We will add a more explicit discussion of this confound and the sensitivity analysis to both the manuscript and the abstract. revision: partial
Referee: Abstract: The 31.5% SPB reduction figure for the cognitive load decomposition strategy is presented without any indication of how it is computed, what the baseline is, or whether it is averaged across all 20 models. The full manuscript must provide the experimental protocol, per-model results, and significance tests supporting this headline number.

Authors: We agree that the abstract presents this number without sufficient context. In the full manuscript, the 31.5% figure is the average relative reduction in SPB across all 20 models, where the baseline is each model's SPB under the standard single-dimension evaluation protocol (i.e., the judge is asked to simply choose the better response). The cognitive load decomposition strategy instead asks the judge to evaluate responses along multiple structured dimensions (e.g., relevance, coherence, completeness, style) before issuing an overall preference, thereby reducing the cognitive load associated with holistic judgment. The per-model results are reported in a table in the experiments section, with paired t-tests showing that the reduction is statistically significant (p < 0.01) for 16 of the 20 models. We also report the standard deviation and confidence intervals. We will revise the abstract to specify that the figure is an average across all 20 models, state the baseline, and note that per-model results and significance tests are provided in the full paper. revision: yes

standing simulated objections not resolved

The referee's review is based on the abstract only, as the full text was not available. Several of the concerns—particularly the lack of methodological detail—are addressed in the full manuscript but cannot be fully conveyed in an abstract. We have summarized the relevant content above and will revise the abstract to better reflect the methodology, but a complete response to some points requires the referee to have access to the full text.

Circularity Check

0 steps flagged

No circularity found: the framework's claims rest on an unverified empirical assumption about equal-quality pair construction, not on a definitional or self-citational reduction.

full rationale

The abstract describes a framework that constructs equal-quality response pairs to measure Self-Preference Bias (SPB) without human gold standards, then proposes a mitigation strategy. The reader's concern—that the automated pair-construction procedure might itself use LLM judgments, potentially confounding the bias measurement—is a legitimate methodological risk, but it is not circularity in the sense defined here. Circularity requires that a claimed derivation reduces by construction to its inputs (e.g., a parameter is fitted to data and then 'predicted' on the same data, or a result is true by definition). Here, the paper makes an empirical claim that equal-quality pairs can be constructed and that SPB can be measured from them. Whether the construction method is sound is a question of correctness and external validity, not circularity. There is no self-citation chain visible in the abstract, no equation that reduces to its own input, and no fitted parameter being renamed as a prediction. The 31.5% bias reduction is an empirical result, not a definitional consequence. Without the full text, no specific reduction can be exhibited, and the abstract alone does not contain circular structure. The score is 1 rather than 0 only because the full text is unavailable, leaving open the possibility that the pair-construction methodology contains a self-referential step not visible in the abstract.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 1 invented entities

The axiom ledger is based solely on the abstract, as the full text is not available. The free parameters and axioms are inferred from the high-level description of the methodology.

free parameters (2)

Equal-quality threshold = Unknown
The framework relies on constructing response pairs with negligible quality differences, which likely requires a threshold parameter to determine what constitutes equal quality. The specific value is not stated in the abstract.
Cognitive load decomposition dimensions = Unknown
The mitigation strategy is grounded in cognitive load decomposition, which likely involves selecting specific dimensions or factors for the structured evaluation. These are not specified in the abstract.

axioms (3)

domain assumption Self-Preference Bias is a directional evaluative deviation that can be isolated from discriminability.
The framework assumes that bias can be statistically disentangled from a model's ability to discriminate quality, which is a core premise of the methodology.
ad hoc to paper Response pairs with negligible quality differences can be constructed without human gold standards.
The automated framework depends on the ability to create equal-quality pairs, which is a load-bearing assumption for the entire measurement approach.
ad hoc to paper Cognitive load decomposition is an effective strategy for mitigating SPB.
The mitigation strategy is grounded in this concept, which is assumed to be applicable to LLM evaluation.

invented entities (1)

Self-Preference Bias (SPB) independent evidence
purpose: To describe the systematic favoring or disfavoring of a model's own generated outputs during evaluation.
The paper provides a falsifiable measurement framework for SPB, giving it independent evidence beyond the paper itself.

pith-pipeline@v1.1.0-glm · 4323 in / 1907 out tokens · 115828 ms · 2026-07-04T17:48:15.444017+00:00 · methodology

0 comments

read the original abstract

LLM-as-a-Judge has become a dominant approach in automated evaluation systems, playing critical roles in model alignment, leaderboard construction, quality control, and so on. However, the scalability and trustworthiness of this approach can be substantially distorted by Self-Preference Bias (SPB), which is a directional evaluative deviation in which LLMs systematically favor or disfavor their own generated outputs during evaluation. Existing measurements rely on costly human annotations and conflate generative capability with evaluative stance, and thus are impractical for large-scale deployment in real-world systems. To address this issue, we introduce a fully automated framework to quantifying and mitigating SPB, which constructs equal-quality pairs of responses with negligible quality differences, enabling statistical disentanglement of discriminability from bias propensity without human gold standards. Empirical analysis across 20 mainstream LLMs reveals that advanced capabilities are often uncorrelated, or even negatively correlated, with low SPB. To mitigate this bias, we propose a structured multi-dimensional evaluation strategy grounded in cognitive load decomposition, which reduces SPB by 31.5\% on average.

Figures

Figures reproduced from arXiv: 2604.22891 by Chuxian Qiu, Jinming Yang, Tao Zhou, Xinshan Jiao, Zheng Hu, Zhenyu Deng.

**Figure 1.** Figure 1: Overview of the SPB quantification and mitigation framework. The workflow view at source ↗

**Figure 2.** Figure 2: Correlation analysis between quality and SPB. The x-axis represents the model’s view at source ↗

**Figure 3.** Figure 3: Discrimination capability (πi) vs. SPB (βi). Dashed lines indicate thresholds: πthresh = 0.8 and |βi |thresh = 0.08. Objective Judges. Three models are good evaluators: DeepSeek-V3- 0324 (DeepSeek-AI, 2024) (π = 0.82, β = 0.024), Grok-4-Fast (π = 0.85, β = 0.035), and Kimi-Linear-48B-A3B-Instruct (Moonshot AI, 2025c) (π = 0.85, β = −0.043). Despite Kimi-Linear-48B-A3B-Instruct’s mild negative bias, all fal… view at source ↗

**Figure 4.** Figure 4: The mitigation effect of the structured multi-dimensional evaluation strategy. view at source ↗

Review history (3 revisions) →

discussion (0)

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Scaffold, Not Vocabulary? A Controlled, Two-Tier, Pre-Registered Study of a Popperian Code-Generation Skill
cs.SE 2026-06 conditional novelty 7.0

Controlled ablation finds Popperian code-generation skill adds no separable correctness benefit over labels-only scaffold; gains track structure not content.
Automated IEP Generation from Traditional Chinese Parent-Teacher Interviews via Corpus-Grounded Feature Diffusion
cs.CL 2026-06 unverdicted novelty 6.0

Corpus-Grounded Feature Diffusion generates synthetic Traditional Chinese IEP training data from 25 seeds to fine-tune a 7B model that reaches BERTScore F1 0.779 on a 10-sample hold-out, beating several larger zero-sh...