pith. sign in

arxiv: 2302.09664 · v3 · pith:C3UQ73X5new · submitted 2023-02-19 · 💻 cs.CL · cs.AI· cs.LG

Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

Lorenz Kuhn , Yarin Gal , Sebastian Farquhar This is my paper

Pith reviewed 2026-05-12 17:55 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.LG
keywords semantic entropyuncertainty estimationnatural language generationquestion answeringlarge language modelssemantic equivalencelinguistic invariancemodel calibration
0
0 comments X

The pith

Semantic entropy, which groups model outputs by shared meaning before measuring uncertainty, predicts answer accuracy more reliably than token-level entropy on question answering tasks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes a way to quantify uncertainty in large language models when they generate natural language answers, such as to questions. Conventional entropy calculations treat every distinct sentence as unique, even when different wordings convey identical information, which distorts the uncertainty signal. To address this, the method first clusters generated sentences into semantic equivalence classes using the model itself without supervision, then computes entropy over the probabilities of these meaning-based clusters. Experiments across question answering datasets demonstrate that this semantic entropy correlates more strongly with whether the model’s answer is correct than several baseline uncertainty measures. The result matters because accurate uncertainty estimates let users know when to trust or disregard a model’s output in practical settings.

Core claim

The authors introduce semantic entropy as an entropy measure over semantic equivalence classes of generated sentences rather than over individual token sequences. Sentences are grouped into classes that share the same meaning through an unsupervised procedure that queries the language model itself; the entropy is then taken with respect to the total probability mass assigned to each class. This construction is invariant to linguistic rephrasings that preserve meaning and requires no model modifications, additional training data, or auxiliary models. Ablation studies on multiple question answering benchmarks show that semantic entropy is more predictive of model accuracy than comparable token

What carries the argument

Semantic entropy: entropy computed over clusters of semantically equivalent generations identified unsupervised by the model itself.

Load-bearing premise

Semantic equivalence classes among generated sentences can be reliably identified in an unsupervised manner using the language model itself.

What would settle it

A dataset or experiment in which the unsupervised clustering places semantically distinct answers into the same class (or vice versa) and semantic entropy loses its advantage in predicting accuracy over baselines.

read the original abstract

We introduce a method to measure uncertainty in large language models. For tasks like question answering, it is essential to know when we can trust the natural language outputs of foundation models. We show that measuring uncertainty in natural language is challenging because of "semantic equivalence" -- different sentences can mean the same thing. To overcome these challenges we introduce semantic entropy -- an entropy which incorporates linguistic invariances created by shared meanings. Our method is unsupervised, uses only a single model, and requires no modifications to off-the-shelf language models. In comprehensive ablation studies we show that the semantic entropy is more predictive of model accuracy on question answering data sets than comparable baselines.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript introduces semantic entropy, an uncertainty measure for natural language generation that incorporates linguistic invariances arising from semantic equivalence among different phrasings. The approach is unsupervised, relies on a single off-the-shelf language model without modifications, and is evaluated via ablation studies claiming superior predictive power for model accuracy on question-answering datasets relative to standard baselines.

Significance. If the central empirical claim holds after addressing the clustering validation, the work would offer a practical advance in uncertainty estimation for NLG by handling semantic equivalence without external supervision or model changes. The unsupervised single-model design is a notable strength that could facilitate broader adoption in reliability-critical applications.

major comments (2)
  1. [Ablation studies] The ablation studies' claim of superior predictive performance for semantic entropy depends on the reliability of the unsupervised semantic equivalence clustering step, yet no details are supplied on the exact prompting/embedding procedure used to form clusters or on any independent validation of cluster quality (e.g., human agreement rates stratified by model confidence level).
  2. [Method] Because equivalence judgments are obtained from the same language model whose uncertainty is being quantified, the clustering step risks producing unreliable or inconsistent partitions precisely when the model is uncertain about the answer; this directly affects the entropy calculation and could inflate the reported advantage over baselines.
minor comments (1)
  1. [Abstract] The abstract states empirical superiority on QA datasets but omits any mention of the statistical tests employed or controls for confounding factors such as generation length or sampling temperature.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments, which help clarify important aspects of our work on semantic entropy. We address each major point below and will revise the manuscript to improve transparency and robustness.

read point-by-point responses

  1. Referee: [Ablation studies] The ablation studies' claim of superior predictive performance for semantic entropy depends on the reliability of the unsupervised semantic equivalence clustering step, yet no details are supplied on the exact prompting/embedding procedure used to form clusters or on any independent validation of cluster quality (e.g., human agreement rates stratified by model confidence level).

    Authors: We agree that greater detail on the clustering procedure is needed for reproducibility. In the revised manuscript, we will expand the methods section to fully specify the prompting strategy for equivalence judgments and the embedding approach used to form clusters. We will also add a human evaluation of cluster quality, reporting agreement rates and stratifying results by model confidence levels to directly validate this component of the method. revision: yes

  2. Referee: [Method] Because equivalence judgments are obtained from the same language model whose uncertainty is being quantified, the clustering step risks producing unreliable or inconsistent partitions precisely when the model is uncertain about the answer; this directly affects the entropy calculation and could inflate the reported advantage over baselines.

    Authors: This is a substantive methodological concern. Using the same model for equivalence judgments introduces a potential dependency that could affect cluster reliability in low-confidence regimes. We will add a dedicated discussion section in the revision addressing this limitation, including analysis of how the entropy measure behaves under varying confidence levels and why the observed performance gains are not solely attributable to this effect. revision: partial

Circularity Check

0 steps flagged

No significant circularity detected in semantic entropy derivation

full rationale

The paper defines semantic entropy by extending standard entropy to group generations into semantic equivalence classes identified unsupervised via the same model. No equations or steps in the provided text reduce the final measure to a fitted parameter, self-referential definition, or load-bearing self-citation by construction. The method is explicitly described as model-agnostic and unsupervised without modifications, and ablation results are presented as empirical comparisons to baselines rather than forced outcomes. This satisfies the default expectation of a non-circular paper; the clustering step is a methodological choice whose quality is not shown to be tautological with the uncertainty output.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review yields no information on free parameters, axioms, or invented entities; the method is described at a high level without implementation specifics.

pith-pipeline@v0.9.0 · 5405 in / 959 out tokens · 42986 ms · 2026-05-12T17:55:04.481439+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. REALISTA: Realistic Latent Adversarial Attacks that Elicit LLM Hallucinations

    cs.CL 2026-05 unverdicted novelty 8.0

    REALISTA optimizes continuous combinations of valid editing directions in latent space to produce realistic adversarial prompts that elicit hallucinations more effectively than prior methods, including on large reason...

  2. An Empirical Study of Security Calibration in Large Language Models for Code

    cs.SE 2026-06 unverdicted novelty 7.0

    Empirical evaluation of three LLMs finds prevalent overconfidence in insecure code generation, with security calibration outperforming functional calibration but both degrading in repository-level settings.

  3. Delayed Verification Destabilizes Multi-Agent LLM Belief: Instability Thresholds and Optimal Corrector Placement

    cs.MA 2026-06 unverdicted novelty 7.0

    Models delayed verification in multi-agent LLMs as graph consensus, derives stability thresholds (inverse golden ratio for delay two) via grounded Laplacian, and gives a supermodular greedy rule for corrector placemen...

  4. SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs

    cs.CV 2026-06 unverdicted novelty 7.0

    SPOT-E uses entropy shaping on answer predictions with low-entropy anchors to optimize visual spotlights at test time via GRPO for better VLM performance on evidence-intensive tasks.

  5. MortarBench: Evaluating Mortgage Loan Origination Agents

    cs.LG 2026-06 unverdicted novelty 7.0

    MortarBench benchmark shows LLMs achieve ≤77.1% accuracy on loan origination; CRIT calibration raises accuracy to 80.5% and reduces bias.

  6. ActProbe: Action-Space Probe for Early Failure Detection of Generative Robot Policies

    cs.RO 2026-06 unverdicted novelty 7.0

    ActProbe is an action-space detector that uses temporal consistency error and action chunk magnitude from policy outputs, mapped via LSTM-MLP, to predict failures earlier than baselines across policies and real-robot tasks.

  7. Remember with Confidence: Uncertainty Quantification for Spatio-temporal Memory with Probabilistic Guarantees

    cs.CV 2026-06 unverdicted novelty 7.0

    Introduces object-level semantic uncertainty for VLM memory, the UQ-DAAAM refinement system, and probabilistic guarantees that selected high-quality views reduce uncertainty more effectively.

  8. Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning

    cs.CL 2026-06 unverdicted novelty 7.0

    Chunk-Level Guided Generation uses off-the-shelf large LLMs to score fixed-length chunks from small models via likelihoods, matching trained PRM performance on math benchmarks without reward-model training.

  9. Before and After Temperature: A Distributional View of Creative LLM Generation

    cs.CL 2026-05 unverdicted novelty 7.0

    A per-token feature from temperature-induced changes in LLM token distributions predicts within-prompt creativity rank at Spearman rho 0.918 vs LLM judges and 0.870 vs humans, outperforming perplexity, entropy, top-1 ...

  10. Not All Uncertainty Is Equal: How Uncertainty Granularity Shapes Human Verification in LLM-Assisted Decision Making

    cs.HC 2026-05 unverdicted novelty 7.0

    A between-subjects experiment (N=192) finds that token-level uncertainty increases agreement with LLM answers while relation-level uncertainty reduces external verification in medical decision tasks.

  11. Proper Scoring Rules for Agentic Uncertainty Quantification

    cs.AI 2026-05 unverdicted novelty 7.0

    Introduces Trajectory Proper Score (TPS) as a strictly proper family of trajectory-level scoring rules that elicits the complete prefix-conditioned success probability process.

  12. CaMo: Camera Motion Grounded Evaluation and Training for Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 7.0

    Proposes Spatial Narrative Score (SNS) evaluation for VLMs' camera motion understanding and introduces CaMo model achieving consistent performance on SNS and direct QA.

  13. Inducing Artificial Uncertainty in Language Models

    cs.CL 2026-05 unverdicted novelty 7.0

    Inducing artificial uncertainty on trivial tasks allows training probes that achieve higher calibration on hard data than standard approaches while retaining performance on easy data.

  14. Utility-Oriented Visual Evidence Selection for Multimodal Retrieval-Augmented Generation

    cs.CL 2026-05 unverdicted novelty 7.0

    Evidence utility is defined as information gain on the model's output distribution, with ranking by gain on a latent helpfulness variable shown equivalent to answer-space utility under mild assumptions, enabling a tra...

  15. Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking

    cs.CL 2026-05 unverdicted novelty 7.0

    BICR trains a lightweight probe on contrastive hidden states from real versus blind images to detect visual grounding in LVLM predictions, outperforming baselines on calibration and discrimination with fewer parameters.

  16. Grounded or Guessing? LVLM Confidence Estimation via Blind-Image Contrastive Ranking

    cs.CL 2026-05 unverdicted novelty 7.0

    BICR uses blind-image contrastive ranking on frozen LVLM hidden states to train a lightweight probe that penalizes confidence on blacked-out inputs, yielding top calibration and discrimination across five models and m...

  17. Task-Aware Calibration: Provably Optimal Decoding in LLMs

    cs.LG 2026-05 unverdicted novelty 7.0

    Task calibration aligns LLM distributions in latent task spaces to make MBR decoding provably optimal and improve generation quality.

  18. Active Testing of Large Language Models via Approximate Neyman Allocation

    cs.AI 2026-05 unverdicted novelty 7.0

    Proposes surrogate semantic entropy stratification followed by approximate Neyman allocation for active testing of LLMs on generative benchmarks, reporting up to 28% MSE reduction and 22.9% average budget savings vers...

  19. Elicitation Matters: How Prompts and Query Protocols Shape LLM Surrogates under Sparse Observations

    cs.CL 2026-05 unverdicted novelty 7.0

    LLM surrogate beliefs under sparse observations depend on prompts and query protocols, with structural prompts as priors, pointwise vs joint querying producing different beliefs, and sequential evidence causing non-mo...

  20. Are LLM Uncertainty and Correctness Encoded by the Same Features? A Functional Dissociation via Sparse Autoencoders

    cs.LG 2026-04 unverdicted novelty 7.0

    Uncertainty and correctness in LLMs are encoded by distinct feature populations, with suppression of confounded features improving accuracy and reducing entropy.

  21. Beyond One Output: Visualizing and Comparing Distributions of Language Model Generations

    cs.AI 2026-04 conditional novelty 7.0

    GROVE visualizes distributions of language model generations as overlapping paths through a text graph, with user studies showing that graph summaries aid structural judgments like diversity assessment while raw outpu...

  22. Structural Anchors and Reasoning Fragility:Understanding CoT Robustness in LLM4Code

    cs.SE 2026-04 unverdicted novelty 7.0

    CoT prompting in LLM4Code shows mixed robustness that depends on model family, task structure, and perturbations destabilizing structural anchors, leading to trajectory deformations like lengthening, branching, and si...

  23. Bridging the Missing-Modality Gap: Improving Text-Only Calibration of Vision Language Models

    cs.CL 2026-04 unverdicted novelty 7.0

    A new Latent Imagination Module uses cross-attention to predict latent visual embeddings from text, improving accuracy and calibration of vision-language models on text-only inputs.

  24. CALIBER: Calibrating Confidence Before and After Reasoning in Language Models

    cs.CL 2026-06 unverdicted novelty 6.0

    CALIBER elicits and supervises pre-reasoning confidence with prompt-level success probability and post-reasoning confidence with answer-level correctness, cutting ECE by 52.5% on BigMathDigits for a 7B model while rem...

  25. The Score Granularity Gap in Black-Box LLM Classification: A Comparative Study of Confidence Constructions

    cs.CL 2026-06 unverdicted novelty 6.0

    Comparative evaluation of seven confidence constructions across 25 LLM-dataset pairs reveals that verbalized scores provide good ranking but coarse granularity for thresholding, while multi-query aggregation helps wea...

  26. Right Knowledge, Wrong Answer: Test-Time Steering for Temporal Fact Conflicts in Open-Weight Language Models

    cs.LG 2026-06 unverdicted novelty 6.0

    Temporal Attractor Steering resolves 29-57% of parametric temporal conflicts in open-weight LLMs while preserving 85-99% accuracy on non-conflict queries.

  27. Confidence Laundering in Agent Systems: Why Uncertainty Needs a Latent Carrier

    cs.AI 2026-06 unverdicted novelty 6.0

    Agent systems lose uncertainty at decision handoffs, causing downstream over-trust; the paper proposes latent uncertainty as a carrier to preserve pre-commitment fragility across interfaces.

  28. FASE: Fast Adaptive Semantic Entropy for Code Quality

    cs.SE 2026-06 unverdicted novelty 6.0

    FASE approximates functional correctness via MST on structural and semantic dissimilarity graphs, reporting 25% better Spearman correlation and 19% better ROCAUC than LLM-based semantic entropy at 0.3% runtime cost on...

  29. CRAFT: Cost-aware Refinement And Front-aware Tuning of Prompts

    cs.CL 2026-06 unverdicted novelty 6.0

    CRAFT is a Pareto-front prompt optimizer that allocates scarce LLM validation calls to candidates near the current front using accuracy- and cost-oriented generators plus NSGA-II retention.

  30. Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVR

    cs.LG 2026-06 unverdicted novelty 6.0

    RLVR exhibits correct-set turnover where solved problems regress during training, and a periodic review mechanism exploiting a repair-window principle improves retention and performance over baselines.

  31. TriLens: Per-Layer Logit-Lens Entropy for White-Box Hallucination Detection

    cs.AI 2026-05 unverdicted novelty 6.0

    TriLens detects hallucinations via per-layer entropy trajectories of logit-lens readouts from three internal modules across LLMs and QA benchmarks.

  32. Functional Entropy: Predicting Functional Correctness in LLM-Generated Code with Uncertainty Quantification

    cs.CL 2026-05 unverdicted novelty 6.0

    Introduces functional equivalence methods and functional entropy to predict functional correctness of LLM-generated code via uncertainty quantification, outperforming NLI-based baselines in most tested settings.

  33. Reading Calibrated Uncertainty from Language Model Trajectories

    cs.LG 2026-05 unverdicted novelty 6.0

    Geometric features from per-layer MLP update trajectories fed to a sparse linear probe outperform maximum softmax probability for uncertainty quantification under selective abstention, with gains up to 21 AURC points.

  34. BalanceRAG: Joint Risk Calibration for Cascaded Retrieval-Augmented Generation

    cs.CL 2026-05 unverdicted novelty 6.0

    BalanceRAG uses sequential graphical testing on a 2D lattice of threshold pairs to certify safe operating points that meet target risk levels in cascaded RAG while increasing coverage.

  35. LLMs as Implicit Imputers: Uncertainty Should Scale with Missing Information

    stat.ML 2026-05 unverdicted novelty 6.0

    Response entropy in LLMs rises with missing context on SQuAD while sampling-based confidence stays high, supporting the multiple imputation criterion and introducing a diagnostic for uncertainty reduction by context level.

  36. REALISTA: Realistic Latent Adversarial Attacks that Elicit LLM Hallucinations

    cs.CL 2026-05 unverdicted novelty 6.0

    REALISTA generates semantically coherent adversarial prompts via latent-space optimization over input-dependent editing directions, achieving stronger hallucination elicitation than prior realistic attacks on open-sou...

  37. Uncertainty Quantification for LLM-based Code Generation

    cs.SE 2026-05 unverdicted novelty 6.0

    RisCoSet applies multiple hypothesis testing to construct risk-controlling partial-program prediction sets for LLM code generation, achieving up to 24.5% less code removal than prior methods at equivalent risk levels.

  38. Active Testing of Large Language Models via Approximate Neyman Allocation

    cs.AI 2026-05 unverdicted novelty 6.0

    Active testing via surrogate semantic entropy stratification and approximate Neyman allocation reduces MSE by up to 28% versus uniform sampling and saves about 23% of the labeling budget on language and multimodal benchmarks.

  39. Annotations Mitigate Post-Training Mode Collapse

    cs.CL 2026-05 unverdicted novelty 6.0

    Annotation-anchored training reduces semantic diversity collapse in post-trained language models by a factor of six compared to standard supervised fine-tuning while preserving instruction-following and improving with scale.

  40. The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations

    cs.AI 2026-05 unverdicted novelty 6.0

    Temporal knowledge drift is encoded as a geometrically orthogonal direction in LLM residual streams, independent of correctness and uncertainty.

  41. Measuring Black-Box Confidence via Reasoning Trajectories: Geometry, Coverage, and Verbalization

    cs.AI 2026-05 unverdicted novelty 6.0

    Trajectory geometry in embedding space fused with coverage and verbalization yields better black-box CoT confidence estimation than self-consistency at lower sample counts across six benchmark-reasoner pairs.

  42. Estimating the Black-box LLM Uncertainty with Distribution-Aligned Adversarial Distillation

    cs.CL 2026-05 unverdicted novelty 6.0

    DisAAD trains a 1%-sized proxy model via adversarial distillation to quantify uncertainty in black-box LLMs by aligning with their output distributions.

  43. Geometry-Calibrated Conformal Abstention for Language Models

    cs.CL 2026-04 unverdicted novelty 6.0

    Geometry-calibrated conformal abstention lets language models abstain from uncertain queries with finite-sample guarantees on both participation rate and conditional correctness of answers.

  44. Trace-Level Analysis of Information Contamination in Multi-Agent Systems

    cs.AI 2026-04 unverdicted novelty 6.0

    Agent workflows can diverge substantially from contaminated inputs yet recover correct answers, or stay similar while failing, as measured by trace divergence on GAIA tasks.

  45. The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive

    cs.CR 2026-04 unverdicted novelty 6.0

    LLM token rank-frequency distributions converge to a shared Mandelbrot distribution across models and domains, enabling a microsecond-scale statistical primitive for provenance verification and black-box anomaly triage.

  46. Detecting Hallucinations for Large Language Model-based Knowledge Graph Reasoning

    cs.CL 2026-04 unverdicted novelty 6.0

    LUCID detects hallucinations in LLM-KG reasoning by extracting node/edge features from attention and semantics then integrating them with KG structure in a GNN, achieving SOTA on nine new benchmark datasets versus 15 ...

  47. Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification

    cs.AI 2026-04 unverdicted novelty 6.0

    Cross-model semantic disagreement adds an epistemic uncertainty term that improves total uncertainty estimation over self-consistency alone, helping flag confident errors in LLMs.

  48. CLSGen: A Dual-Head Fine-Tuning Framework for Joint Probabilistic Classification and Verbalized Explanation

    cs.CL 2026-04 unverdicted novelty 6.0

    CLSGen is a dual-head LLM fine-tuning framework that enables joint probabilistic classification and verbalized explanation generation without catastrophic forgetting of generative capabilities.

  49. Rag Performance Prediction for Question Answering

    cs.CL 2026-04 unverdicted novelty 6.0

    A novel supervised predictor modeling semantic relationships among question, retrieved passages, and generated answer best forecasts when RAG improves QA performance.

  50. Ensemble-Based Uncertainty Estimation for Code Correctness Estimation

    cs.SE 2026-03 unverdicted novelty 6.0

    Ensemble Semantic Entropy improves correlation with code correctness over single-model methods and powers a cascading scaling system that cuts FLOPs by 64.9% while preserving performance on LiveCodeBench.

  51. ETS: Energy-Guided Test-Time Scaling for Training-Free RL Alignment

    cs.LG 2026-01 conditional novelty 6.0

    ETS enables direct sampling from the optimal RL policy for language models at inference time by estimating the energy term with online Monte Carlo and acceleration techniques.

  52. ETS: Energy-Guided Test-Time Scaling for Training-Free RL Alignment

    cs.LG 2026-01 unverdicted novelty 6.0

    ETS performs training-free RL alignment for language models by energy-guided test-time scaling with Monte Carlo energy estimation and importance sampling acceleration.

  53. A Geometric Taxonomy of Hallucinations in LLMs

    cs.AI 2026-01 unverdicted novelty 6.0

    Embedding geometry on the unit hypersphere distinguishes detectable query-proximate unfaithfulness and confabulations from undetectable factual errors sharing vocabulary with correct answers.

  54. GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughts

    cs.AI 2026-01 unverdicted novelty 6.0

    GlimpRouter uses the entropy of the first token in each reasoning step to decide whether to invoke a large model, yielding 10.7% higher accuracy and 25.9% lower latency than a standalone large model on AIME25.

  55. Principled Detection of Hallucinations in Large Language Models via Multiple Testing

    cs.CL 2025-08 unverdicted novelty 6.0

    The method aggregates multiple hallucination evaluation scores via conformal p-values to enable calibrated detection with controlled false alarm rates across LLMs and datasets.

  56. Neuron-Aware Active Few-Shot Learning for LLMs

    cs.LG 2026-07 unverdicted novelty 5.0

    NeuFS selects active few-shot samples for LLMs by representing samples via neuron activation patterns and applying a dual-criteria strategy of diversity and neuron consensus to identify informative examples.

  57. Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimation

    cs.LG 2026-07 unverdicted novelty 5.0

    DALorRA applies variational Bayesian sparse masking to LoRA ranks to calibrate LLM uncertainty while preserving accuracy.

  58. Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling

    cs.AI 2026-07 unverdicted novelty 5.0

    C3RL is a new RL algorithm combining correctness, calibration, and reference accuracy rewards to improve LLM confidence calibration, enabling CAS to outperform majority voting with up to 12.33x lower inference cost.

  59. From Signals to Transfer: A Factorised Study of Probe-Based Uncertainty Estimation in Large Language Models

    cs.CL 2026-06 conditional novelty 5.0

    A factorized study finds raw hidden states and attention features hard to beat in-domain for LLM uncertainty probes, but structured compressed features are more robust under distribution shift, with pretrained probes ...

  60. EviRank: Evidence-Based Confidence Estimation for LLM-Based Ranking

    cs.IR 2026-06 unverdicted novelty 5.0

    EviRank extracts three evidences from a single LLM forward pass, aggregates them with reliable opinion pooling and position-aware calibration, then uses the result to optimize rankings, claiming SOTA on recommendation...

Reference graph

Works this paper leans on

27 extracted references · 27 canonical work pages · cited by 77 Pith papers · 7 internal anchors

  1. [1]

    Language models are few-shot learners

    6 Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

    work page 1901

  2. [2]

    PaLM: Scaling Language Modeling with Pathways

    1 Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    Calibration of pre-trained transformers

    6 Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pp. 295– 302,

    work page 2020

  4. [4]

    Unsupervised quality estimation for neural machine translation

    7 10 Published as a conference paper at ICLR 2023 Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Fr´ed´eric Blain, Francisco Guzm´an, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics , 8: 539–555,

    work page 2023

  5. [5]

    Uncertainty-aware ma- chine translation evaluation

    1, 2 Taisiya Glushkova, Chrysoula Zerva, Ricardo Rei, and Andr ´e FT Martins. Uncertainty-aware ma- chine translation evaluation. arXiv preprint arXiv:2109.06352,

    work page arXiv

  6. [6]

    DeBERTa: Decoding-enhanced BERT with Disentangled Attention

    6 Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654, 2020a. 5 Ruining He, Anirudh Ravula, Bhargav Kanagal, and Joshua Ainslie. Realformer: Transformer likes residual attention. arXiv preprint arXiv:2012.11747, 2020b. 6 Dan Hendrycks, Nicholas Carlini, Joh...

    work page internal anchor Pith review arXiv 2006

  7. [7]

    Training Compute-Optimal Large Language Models

    1 Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203.15556,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Abstract meaning repre- sentation for paraphrase detection

    1 Fuad Issa, Marco Damonte, Shay B Cohen, Xiaohui Yan, and Yi Chang. Abstract meaning repre- sentation for paraphrase detection. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long Papers), pp. 442–452,

    work page 2018

  9. [9]

    Deup: Direct epistemic uncertainty prediction

    6 Moksh Jain, Salem Lahlou, Hadi Nekoei, Victor Butoi, Paul Bertin, Jarrid Rector-Brooks, Maksym Korablyov, and Yoshua Bengio. Deup: Direct epistemic uncertainty prediction. arXiv preprint arXiv:2102.08501,

    work page arXiv

  10. [10]

    TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

    6 Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Language Models (Mostly) Know What They Know

    7 Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language mod- els (mostly) know what they know. arXiv preprint arXiv:2207.05221,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    Teaching Models to Express Their Uncertainty in Words

    7 Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334, 2022a. 2, 3, 6 11 Published as a conference paper at ICLR 2023 Zi Lin, Jeremiah Zhe Liu, and Jingbo Shang. Towards collaborative neural-symbolic graph semantic parsing via uncertainty. In Findings of the Association for C...

    work page internal anchor Pith review arXiv 2023

  13. [13]

    arXiv preprint (2020), https://arxiv.org/abs/2002.07650, arXiv:2002.07650

    3 Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. arXiv preprint arXiv:2002.07650,

    work page arXiv 2002

  14. [14]

    Regression Prior Networks, December 2020

    1, 2, 3, 4, 5, 7, 8 Andrey Malinin, Sergey Chervontsev, Ivan Provilkov, and Mark Gales. Regression prior networks. arXiv preprint arXiv:2006.11590,

    work page arXiv 2006

  15. [15]

    Linguistic calibration through metacognition: aligning dialogue agent responses with expected correctness

    3 Sabrina J Mielke, Arthur Szlam, Y-Lan Boureau, and Emily Dinan. Linguistic calibration through metacognition: aligning dialogue agent responses with expected correctness. arXiv preprint arXiv:2012.14983,

    work page arXiv 2012

  16. [16]

    Correcting Length Bias in Neural Machine Translation

    6 Kenton Murray and David Chiang. Correcting length bias in neural machine translation. arXiv preprint arXiv:1808.10006,

    work page Pith review arXiv

  17. [17]

    Charformer: Fast character transformers via gradient-based subword tokenization.arXiv preprint arXiv:2106.12672, 2021

    3 Yi Tay, Vinh Q Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Si- mon Baumgartner, Cong Yu, and Donald Metzler. Charformer: Fast character transformers via gradient-based subword tokenization. arXiv preprint arXiv:2106.12672,

    work page arXiv

  18. [18]

    Alex Warstadt, Amanpreet Singh, and Samuel R Bowman

    6 Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. Entailment as few-shot learner. arXiv preprint arXiv:2104.14690,

    work page arXiv

  19. [19]

    Bilateral Multi-Perspective Matching for Natural Language Sentences

    6 Zhiguo Wang, Wael Hamza, and Radu Florian. Bilateral multi-perspective matching for natural language sentences. arXiv preprint arXiv:1702.03814,

    work page Pith review arXiv

  20. [20]

    A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

    6 Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426,

    work page Pith review arXiv

  21. [21]

    Deep Learning for Answer Sentence Selection

    5 12 Published as a conference paper at ICLR 2023 Lei Yu, Karl Moritz Hermann, Phil Blunsom, and Stephen Pulman. Deep learning for answer sentence selection. arXiv preprint arXiv:1412.1632,

    work page Pith review arXiv 2023

  22. [22]

    OPT: Open Pre-trained Transformer Language Models

    6 Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068,

    work page internal anchor Pith review Pith/arXiv arXiv

  23. [23]

    Work with foundation mod- els implicitly focuses on lexical equivalence, which entails the others, but we usually care about semantic equivalence

    2, 7 13 Published as a conference paper at ICLR 2023 Table 3: Illustration of semantic, syntactic, and lexical equivalence. Work with foundation mod- els implicitly focuses on lexical equivalence, which entails the others, but we usually care about semantic equivalence. Equivalence Sentence A Sentence B Lexical Syntactic Semantic Paris is the capital of F...

    work page 2023

  24. [24]

    <g/>”,x, s(m))) ⊿ Does old sequence entail new one? right ← M(cat(x, s(m), “<g/>

    Lexically equivalent sequences use exactly the same symbols. They are always also semantically and syntactically equiv- alent (in a given context). Syntactically equivalent sentences have the same grammatical form. But they can have different meanings (not semantically equivalent) and can use different symbols (not lexically equivalent). Semantically equi...

    work page 2023

  25. [25]

    Additionally, we investigate, why the semantic entropy underperforms the length-normalised entropy at high temperatures

    As in the main body of the paper, we measure diversity as the average lexical overlap of the answers in the answer set. Additionally, we investigate, why the semantic entropy underperforms the length-normalised entropy at high temperatures. To that end, we manually inspect and label 100 classifications of our semantic equivalence method at T=1.5, and we fin...

    work page 2023

  26. [26]

    We find that on CoQA, we obtain accurate model results with zero-shot prompting

    We use the following prompts on CoQA and TriviaQA. We find that on CoQA, we obtain accurate model results with zero-shot prompting. While we have to use few-shot prompting to obtain accurate answers on closed-book TriviaQA. We use the following prompts for each of the settings: CoQA: [The provided context paragraph] [additional question-answer pairs] Q: [P...

    work page 2023

  27. [27]

    brainstormed answers

    except for the exact matching accuracy criterion which is too demanding because of the much larger variety of possible answers for this task. 17 Published as a conference paper at ICLR 2023 Table 7: CoQA: the exact choice of the accuracy metric for the free-form open-book QA task has little effect on the assessment of the quality of the uncertainty measur...

    work page 2023