Pith. sign in

REVIEW 2 major objections 2 minor 101 references

ECUAS_n is a family of proper scoring rules that evaluate uncertainty-augmented systems by balancing the cost of wrong predictions against the quality of uncertainty estimates through a single tunable parameter n.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 17:52 UTC pith:IP7H37WA

load-bearing objection ECUAS_n offers a parameterized proper scoring rule family for joint evaluation of predictions and uncertainty, but the abstract gives no derivations so the claims stay unverified. the 2 major comments →

arxiv 2605.20490 v3 pith:IP7H37WA submitted 2026-05-19 cs.AI cs.LG

ECUAS_n: A family of metrics for principled evaluation of uncertainty-augmented systems

Lautaro Estienne , Erik Ernst , Mat\'ias Vera , Pablo Piantanida , Luciana Ferrer This is my paper
classification cs.AI cs.LG
keywords uncertainty evaluationproper scoring rulesuncertainty-augmented systemsdecision making under uncertaintyclassification tasksgeneration tasksrejection decisionscost trade-offs
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Current evaluation methods assess predictions and uncertainty estimates in isolation, apply a single fixed rejection cost, or rely on coverage-risk curves, all of which the paper treats as insufficient for judging how well a system supports real decisions under uncertainty. The authors introduce ECUAS_n as a family of metrics cast directly as proper scoring rules for the underlying task. The parameter n sets the relative weight between paying for prediction mistakes and paying for imperfect uncertainty information, so the same metric family can be adapted to different application cost structures. Experiments across classification and generation datasets, including a hand-labeled TriviaQA subset, illustrate the metrics in practice. A reader would care because high-stakes automated decisions require an evaluation method that directly rewards systems whose uncertainties help users or downstream processes accept or reject outputs at the right times.

Core claim

The paper claims that ECUAS_n metrics, defined as proper scoring rules for the decision task, give a unified assessment of uncertainty-augmented systems. The single parameter n directly controls the trade-off between the cost of incorrect predictions and the cost of imperfect uncertainties according to use-case needs, replacing separate metrics or fixed rejection costs. Advantages are shown both by the proper-scoring formulation itself and by empirical comparisons on diverse classification and generation data.

What carries the argument

ECUAS_n, the family of metrics expressed as proper scoring rules for the task of interest, where the parameter n sets the relative penalty for prediction errors versus uncertainty shortfalls.

Load-bearing premise

Existing approaches that evaluate predictions and uncertainty scores separately or with a fixed rejection cost are inadequate for assessing the overall performance of uncertainty-augmented systems for decision making under uncertainty.

What would settle it

A controlled decision-making trial in which systems are ranked by ECUAS_n at different n values and the ranking is checked against measured total decision cost (or utility) when users actually accept or reject predictions according to the reported uncertainties.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single metric can now rank uncertainty-augmented systems according to how well their uncertainties support decisions at application-specific cost ratios.
  • Choosing different values of n produces different orderings that reflect different relative costs of mistakes versus uncertainty quality.
  • Because the metrics are proper scoring rules, reporting better-calibrated uncertainties for the chosen n directly improves the score.
  • Empirical results on classification and generation tasks demonstrate that the metrics distinguish system performance in ways prior separate or fixed-cost methods do not.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same family could be applied to regression or structured prediction tasks by re-expressing the underlying proper scoring rule for those outputs.
  • Model selection or training objectives could be defined directly in terms of ECUAS_n at a target n to align optimization with downstream decision costs.
  • The approach connects evaluation to expected-utility decision theory, suggesting that the metrics may correlate more closely with realized user or system utility than isolated calibration or accuracy numbers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript argues that separate evaluation of predictions and uncertainties, fixed rejection costs, or coverage-risk curves are inadequate for UA systems in decision-making under uncertainty. It proposes the ECUAS_n family of metrics, formulated as proper scoring rules where the tunable parameter n trades off the cost of incorrect predictions against imperfect uncertainty estimates. Theoretical advantages and empirical results are presented on classification and generation tasks, including a manually annotated TriviaQA subset.

Significance. If the proper-scoring-rule derivation holds and the empirical comparisons are robust, ECUAS_n would supply a unified, application-tunable evaluation framework that directly supports cost-sensitive decisions. The explicit use of proper scoring rules and the inclusion of generation tasks are strengths that could improve reproducibility and comparability across UA-system papers.

major comments (2)
  1. [§3] §3 (theoretical formulation): the manuscript must explicitly derive that ECUAS_n satisfies the proper scoring rule property (expected score minimized exactly when the reported distribution matches the true conditional) for arbitrary n; without this step-by-step verification the central claim that the metric is 'formulated as proper scoring rules' cannot be assessed.
  2. [§4] §4 (experiments): the data-exclusion rules, exact definition of the 'manually annotated subset of TriviaQA', and the precise implementation of the baseline metrics (separate evaluation, fixed-cost, coverage-risk) are not stated with sufficient precision to determine whether post-hoc choices affect the reported advantages.
minor comments (2)
  1. [§2] Notation for the uncertainty score and the rejection decision should be introduced once in a dedicated subsection rather than piecemeal.
  2. [Figures 2-4] Figure captions should state the exact value of n used in each panel and whether error bars reflect multiple random seeds or bootstrap replicates.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which highlight areas where additional rigor and detail will strengthen the manuscript. We address each major comment below and will revise the paper to incorporate the requested clarifications and derivations.

read point-by-point responses

  1. Referee: [§3] §3 (theoretical formulation): the manuscript must explicitly derive that ECUAS_n satisfies the proper scoring rule property (expected score minimized exactly when the reported distribution matches the true conditional) for arbitrary n; without this step-by-step verification the central claim that the metric is 'formulated as proper scoring rules' cannot be assessed.

    Authors: We agree that an explicit, step-by-step derivation is essential for substantiating the proper scoring rule claim. In the revised §3 we will add a self-contained proof that, for any fixed n, the expected value of ECUAS_n is uniquely minimized when the reported distribution equals the true conditional distribution. The derivation will start from the definition of ECUAS_n, apply the law of total expectation, and show that the resulting expression is a strictly proper scoring rule by direct comparison to the true distribution (extending the n=1 case already sketched in the current text). revision: yes

  2. Referee: [§4] §4 (experiments): the data-exclusion rules, exact definition of the 'manually annotated subset of TriviaQA', and the precise implementation of the baseline metrics (separate evaluation, fixed-cost, coverage-risk) are not stated with sufficient precision to determine whether post-hoc choices affect the reported advantages.

    Authors: We acknowledge that the current experimental section lacks the level of detail needed for full reproducibility. In the revision we will expand §4 (and the associated appendix) to: (i) list all data-exclusion criteria with exact thresholds, (ii) specify the annotation protocol, inter-annotator agreement, and final size of the TriviaQA subset, and (iii) provide pseudocode or explicit formulas for the three baseline metrics, including how rejection costs were set and how coverage-risk curves were integrated. These additions will make it possible to verify that the reported advantages are not artifacts of post-hoc decisions. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected in ECUAS_n derivation

full rationale

The paper introduces ECUAS_n as a family of proper scoring rules whose parameter n trades off prediction error against uncertainty quality for uncertainty-augmented systems. The abstract frames this as a direct formulation motivated by limitations of separate or fixed-cost evaluations, without any visible reduction of the metric to a fitted quantity, self-defined parameter, or load-bearing self-citation. No equations are presented that would make the proposed scores equivalent to their inputs by construction, and the claim is positioned as building on the established concept of proper scoring rules rather than deriving from the authors' prior fitted results. The derivation chain therefore remains self-contained and externally verifiable.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract supplies insufficient detail to enumerate free parameters, axioms, or invented entities; the central claim rests on the unexamined assertion that proper scoring rules are the appropriate formalism and that n meaningfully captures the desired trade-off.

pith-pipeline@v0.9.1-grok · 5727 in / 1058 out tokens · 28791 ms · 2026-06-30T17:52:50.778339+00:00 · methodology

0 comments
read the original abstract

In high-stakes automated decision-making, access to predictive uncertainty is essential for enabling users -- human or downstream systems -- to accept or reject predictions based on application-specific cost trade-offs. Such uncertainty-augmented (UA) systems -- i.e., systems that output both predictions and uncertainty scores -- are currently being assessed in the literature in a variety of ways, using separate metrics to evaluate the predictions and the uncertainty scores, setting a cost function with a fixed rejection cost or integrating over a coverage-risk curve. We argue that these evaluation approaches are inadequate for assessing overall performance of the UA system for decision making under uncertainty and propose a novel family of metrics, ECUAS$_n$, formulated as proper scoring rules for the task of interest. The parameter $n$ controls the trade-off between the cost of incorrect predictions and imperfect uncertainties depending on the needs of the use-case. We demonstrate the advantages of the ECUAS$_n$ metrics both theoretically and empirically, through experiments on diverse classification and generation datasets, including a manually annotated subset of TriviaQA.

Figures

Figures reproduced from arXiv: 2605.20490 by Erik Ernst, Lautaro Estienne, Luciana Ferrer, Mat\'ias Vera, Pablo Piantanida.

Figure 1
Figure 1. Figure 1: C ∗ n as a function of the confidence qe, when candidate decisions are correct (solid lines) and incorrect (dashed lines), for different values of n, the parameter in w, and K, the number of classes. 3 Application of ECUAS to generative systems An important family of UA systems is that based on generative models [30, 87, 58]. To use the ECUAS metrics in this scenario, we need to adapt the definition of C˜.… view at source ↗
Figure 2
Figure 2. Figure 2: ECUASn values when temperature scaling is applied to the calibrated version of q and the candidate answer is obtained by sampling from the resulting distribution. Our evaluation spans multiple state-of-the-art small LLMs, Qwen 3.5 (4B and 9B) [75], GLM-4.6V￾Flash [76], Ministral-3-8B-Instruct-2512 [56], as well as larger models from the Gemini 2.5 family (Flash Lite, Flash and Pro) [12]. We evaluate these … view at source ↗

Review history (3 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

101 extracted references · 31 canonical work pages · 5 internal anchors

  1. [1]

    Ashukha, A

    A. Ashukha, A. Lyzhov, D. Molchanov, and D. Vetrov. Pitfalls of in-domain uncertainty estima- tion and ensembling in deep learning. InInternational Conference on Learning Representations,

  2. [2]

    URLhttps://openreview.net/forum?id=BJxI5gHKDr

  3. [3]

    P. L. Bartlett and M. H. Wegkamp. Classification with a reject option using a hinge loss.J. Mach. Learn. Res., 9:1823–1840, 2008. URL https://api.semanticscholar.org/CorpusID: 16963069

    2008

  4. [4]

    Brummer.Measuring, refining and calibrating speaker and language information extracted from speech

    N. Brummer.Measuring, refining and calibrating speaker and language information extracted from speech. PhD thesis, University of Stellenbosch, 2010. URL https://scholar.sun.ac. za/items/1b46805b-2b1e-46aa-83ce-75ede92f0159

    2010

  5. [5]

    Brümmer.Measuring, Refining and Calibrating Speaker and Language Information Ex- tracted from Speech

    N. Brümmer.Measuring, Refining and Calibrating Speaker and Language Information Ex- tracted from Speech. PhD thesis, Stellenbosch University, 2010

    2010

  6. [6]

    T. J. Bungert, L. Kobelke, and P. F. Jaeger. Understanding silent failures in medical image classification. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, pages 400–410. Springer, 2023

    2023

  7. [7]

    and Lee, Sungbok and Narayanan, Shrikanth S

    C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower Provost, S. Kim, J. Chang, S. Lee, and S. Narayanan. Iemocap: Interactive emotional dyadic motion capture database.Language Resources and Evaluation, 42:335–359, 12 2008. doi: 10.1007/s10579-008-9076-6

    work page doi:10.1007/s10579-008-9076-6 2008

  8. [8]

    L. F. P. Cattelan and D. Silva. How to fix a broken confidence estimator: Evaluating post- hoc methods for selective classification with deep neural networks. InThe 40th Conference on Uncertainty in Artificial Intelligence, 2024. URL https://openreview.net/forum?id= IJBWLRCvYX

    2024

  9. [9]

    J. Cen, D. Luan, S. Zhang, Y . Pei, Y . Zhang, D. Zhao, S. Shen, and Q. Chen. The devil is in the wrongly-classified samples: Towards unified open-set recognition.arXiv preprint arXiv:2302.04002, 2023

    work page arXiv 2023

  10. [10]

    Charoenphakdee, Z

    N. Charoenphakdee, Z. Cui, Y . Zhang, and M. Sugiyama. Classification with rejection based on cost-sensitive classification. InInternational Conference on Machine Learning, 2020. URL https://api.semanticscholar.org/CorpusID:225041187

    2020

  11. [11]

    Cheng, X.-Y

    Z. Cheng, X.-Y . Zhang, and C.-L. Liu. Unified classification and rejection: A one-versus-all framework.arXiv preprint arXiv:2311.13355, 2023

    work page arXiv 2023

  12. [12]

    C. K. Chow. An optimum character recognition system using decision functions.IRE Trans- actions on Electronic Computers, EC-6(4):247–254, Dec. 1957. ISSN 0367-9950. doi: 10.1109/TEC.1957.5222035. URLhttps://ieeexplore.ieee.org/document/5222035

    work page doi:10.1109/tec.1957.5222035 1957

  13. [13]

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

    G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: Pushing the frontier with advanced rea- soning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  14. [14]

    A. P. Dawid and M. Musio. Theory and applications of proper scoring rules.METRON, 72(2): 169–183, Apr 2014. ISSN 2281-695X

    2014

  15. [15]

    Y . Ding, J. Liu, J. Xiong, and Y . Shi. Revisiting the evaluation of uncertainty estimation and its application to explore model complexity-uncertainty trade-off. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 4–5, 2020

    2020

  16. [16]

    J. Duan, H. Cheng, S. Wang, A. Zavalny, C. Wang, R. Xu, B. Kailkhura, and K. Xu. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. InProceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics, Bangkok, Thailand, Aug. 2024. URL https://aclanthology.org/2024....

    2024

  17. [18]

    Dyrland, A

    K. Dyrland, A. S. Lundervold, and P. G. L. P. Mana. Does the evaluation stand up to evaluation? a first-principle approach to the evaluation of classifiers, 2023. URL https://arxiv.org/ abs/2302.12006

    work page arXiv 2023

  18. [19]

    El-Yaniv and Y

    R. El-Yaniv and Y . Wiener. On the Foundations of Noise-free Selective Classification.Journal of Machine Learning Research, 11(53):1605–1641, 2010. ISSN 1533-7928. URL http: //jmlr.org/papers/v11/el-yaniv10a.html

    2010

  19. [20]

    Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification

    E. Fadeeva, A. Rubashevskii, A. Shelmanov, S. Petrakov, H. Li, H. Mubarak, E. Tsym- balov, G. Kuzmin, A. Panchenko, T. Baldwin, P. Nakov, and M. Panov. Fact-checking the output of large language models via token-level uncertainty quantification. In L.- W. Ku, A. Martins, and V . Srikumar, editors,Findings of the Association for Compu- tational Linguistics...

    work page doi:10.18653/v1/2024.findings-acl.558 2024

  20. [21]

    Nature630, 625–630 (06 2024)

    S. Farquhar, J. Kossen, L. Kuhn, and Y . Gal. Detecting hallucinations in large language models using semantic entropy.Nature, 630:625–630, 06 2024. doi: 10.1038/s41586-024-07421-0

    work page doi:10.1038/s41586-024-07421-0 2024

  21. [22]

    L. Ferrer. No need for ad-hoc substitutes: The expected cost is a principled all-purpose classification metric.Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URLhttps://openreview.net/forum?id=5PPbvCExZs

    2025

  22. [23]

    Ferrer and D

    L. Ferrer and D. Ramos. Evaluating posterior probabilities: Decision theory, proper scoring rules, and calibration.Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URLhttps://openreview.net/forum?id=qbrE0LR7fF

    2025

  23. [24]

    Franc, D

    V . Franc, D. Prusa, and V . V oracek. Optimal strategies for reject option classifiers.Journal of Machine Learning Research, 24(11):1–49, 2023

    2023

  24. [25]

    Franc, D

    V . Franc, D. Prusa, and V . V oracek. Optimal Strategies for Reject Option Classifiers.Journal of Machine Learning Research, 24(11):1–49, 2023. ISSN 1533-7928. URL http://jmlr.org/ papers/v24/21-0048.html

    2023

  25. [26]

    Galil and R

    I. Galil and R. El-Yaniv. Disrupting deep uncertainty estimation without harming accuracy. Advances in Neural Information Processing Systems, 34:21285–21296, 2021

    2021

  26. [27]

    X. Gao, J. Zhang, L. Mouatadid, and K. Das. SPUQ: Perturbation-based uncertainty quan- tification for large language models. In Y . Graham and M. Purver, editors,Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2336–2346, St. Julian’s, Malta, Mar. 2024. Association f...

    work page doi:10.18653/v1/2024.eacl-long.143 2024

  27. [28]

    Geifman and R

    Y . Geifman and R. El-Yaniv. Selective Classification for Deep Neural Networks. InAdvances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https://papers.nips.cc/paper_files/paper/2017/hash/ 4a8423d5e91fda00bb7e46540e2b0cf1-Abstract.html

    2017

  28. [30]

    Geifman, G

    Y . Geifman, G. Uziel, and R. El-Yaniv. Bias-reduced uncertainty estimation for deep neural classifiers. InInternational Conference on Learning Representations, 2019. URL https: //openreview.net/forum?id=SJfb5jCqKm

    2019

  29. [31]

    J. Geng, F. Cai, Y . Wang, H. Koeppl, P. Nakov, and I. Gurevych. A survey of confidence estimation and calibration in large language models. In K. Duh, H. Gomez, and S. Bethard, editors,Proceedings of the 2024 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p...

    work page doi:10.18653/v1/2024.naacl-long.366 2024

  30. [32]

    URL���������������� ��������������������������

    T. Gneiting and A. E. Raftery. Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477):359–378, Mar. 2007. ISSN 0162- 1459, 1537-274X. doi: 10.1198/016214506000001437. URL http://www.tandfonline.com/ doi/abs/10.1198/016214506000001437

    work page doi:10.1198/016214506000001437 2007

  31. [33]

    I. J. Good. Rational decisions.Journal of the Royal Statistical Society: Series B (Methodologi- cal), 14(1):107–114, 01 1952. ISSN 0035-9246. doi: 10.1111/j.2517-6161.1952.tb00104.x. URLhttps://doi.org/10.1111/j.2517-6161.1952.tb00104.x

    work page doi:10.1111/j.2517-6161.1952.tb00104.x 1952

  32. [34]

    A. Gulli. The anatomy of a news search engine. InSpecial Interest Tracks and Posters of the 14th International Conference on World Wide Web, pages 880–881, New York, 2005

    2005

  33. [35]

    C. Guo, G. Pleiss, Y . Sun, and K. Q. Weinberger. On calibration of modern neural networks. In Proc. of the 34th International Conference on Machine Learning, Sydney, Australia, 2017

    2017

  34. [36]

    W. He, Z. Jiang, T. Xiao, Z. Xu, and Y . Li. A survey on uncertainty quantification methods for deep learning.ACM Comput. Surv., 58(7), Feb. 2026. ISSN 0360-0300. doi: 10.1145/3786319. URLhttps://doi.org/10.1145/3786319

    work page doi:10.1145/3786319 2026

  35. [37]

    A. D. Hendrickson and R. J. Buehler. Proper scores for probability forecasters.The Annals of Mathematical Statistics, pages 1916–1921, 1971

    1916

  36. [38]

    Hendrycks and K

    D. Hendrycks and K. Gimpel. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. Feb. 2017. URL https://openreview.net/forum?id= Hkg4TI9xl

    2017

  37. [39]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Mea- suring massive multitask language understanding. InInternational Conference on Learning Representations, 2021. URLhttps://openreview.net/forum?id=d7KBjmI3GmQ

    2021

  38. [40]

    J. Heo, H. B. Lee, S. Kim, J. Lee, K. J. Kim, E. Yang, and S. J. Hwang. Uncertainty-aware attention for reliable interpretation and prediction. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors,Advances in Neural Information Process- ing Systems, volume 31. Curran Associates, Inc., 2018. URLhttps://proceedings.n...

    2018

  39. [41]

    B. Hou, Y . Liu, K. Qian, J. Andreas, S. Chang, and Y . Zhang. Decomposing uncertainty for large language models through input clarification ensembling. InProceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024

    2024

  40. [42]

    M. G. M. Hunink, M. C. Weinstein, E. Wittenberg, M. F. Drummond, J. S. Pliskin, J. B. Wong, and P. P. Glasziou.Decision Making in Health and Medicine: Integrating Evidence and Values. Cambridge University Press, 2 edition, 2014

    2014

  41. [43]

    P. F. Jäger, C. Lüth, L. Klein, and T. Bungert. A call to reflect on evaluation practices for failure detection in image classification. InICLR 2023, 2023

    2023

  42. [44]

    Jiang, J

    Z. Jiang, J. Araki, H. Ding, and G. Neubig. How can we know when language models know? on the calibration of language models for question answering.Transactions of the Association for Computational Linguistics, 9:962–977, 2021. doi: 10.1162/tacl_a_00407. URL https://aclanthology.org/2021.tacl-1.57/

    work page doi:10.1162/tacl_a_00407 2021

  43. [45]

    Weld and Luke Zettlemoyer , editor =

    M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In R. Barzilay and M.-Y . Kan, editors,Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada, July 2017. Association for Comp...

    work page doi:10.18653/v1/p17-1147 2017

  44. [46]

    Language Models (Mostly) Know What They Know

    S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield- Dodds, N. DasSarma, E. Tran-Johnson, S. Johnston, S. El-Showk, A. Jones, N. Elhage, T. Hume, A. Chen, Y . Bai, S. Bowman, S. Fort, D. Ganguli, D. Hernandez, J. Jacobson, J. Kernion, S. Kravec, L. Lovitt, K. Ndousse, C. Olsson, S. Ringer, D. Amodei, T. Brown, J. ...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  45. [47]

    Kahneman.Thinking, fast and slow

    D. Kahneman.Thinking, fast and slow. 1st ed. New York : Farrar, Straus and Giroux, 2011. URLhttps://search.library.wisc.edu/catalog/9910114919702121. 12

    work page arXiv 2011

  46. [48]

    Kapoor, N

    S. Kapoor, N. Gruver, M. Roberts, A. Pal, S. Dooley, M. Goldblum, and A. Wilson. Calibration- tuning: Teaching large language models to know what they don’t know. In R. Vázquez, H. Celikkanat, D. Ulmer, J. Tiedemann, S. Swayamdipta, W. Aziz, B. Plank, J. Baan, and M.-C. de Marneffe, editors,Proceedings of the 1st Workshop on Uncertainty-Aware NLP (Uncerta...

    work page doi:10.18653/v1/2024.uncertainlp-1.1 2024

  47. [49]

    J. Kim, J. Koo, and S. Hwang. A unified benchmark for the unknown detection capability of deep neural networks.Expert Systems with Applications, 229:120461, 2023

    2023

  48. [50]

    Krizhevsky and G

    A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009. URL https://www.cs.toronto. edu/~kriz/learning-features-2009-TR.pdf

    2009

  49. [51]

    L. Kuhn, Y . Gal, and S. Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. InThe Eleventh International Conference on Learning Representations, 2023. URLhttps://openreview.net/forum?id=VD-AYtP0dve

    2023

  50. [52]

    Calibration of Encoder Decoder Models for Neural Machine Translation

    A. Kumar and S. Sarawagi. Calibration of encoder decoder models for neural machine transla- tion.arXiv preprint arXiv:1903.00802, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1903

  51. [53]

    Lakshminarayanan, A

    B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc...

    2017

  52. [54]

    S. Lin, J. Hilton, and O. Evans. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research, 2022. URL https://openreview.net/forum? id=8s8K2UZGTZ

    2022

  53. [55]

    Z. Lin, S. Trivedi, and J. Sun. Generating with confidence: Uncertainty quantification for black-box large language models.Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URLhttps://openreview.net/forum?id=DWkJCSxKU5

    2024

  54. [56]

    Z. Lin, S. Trivedi, and J. Sun. Contextualized sequence likelihood: Enhanced confidence scores for natural language generation. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA, Nov. 2024. URL https: //aclanthology.org/2024.emnlp-main.578/

    2024

  55. [57]

    A. Liu, K. Khandelwal, S. Subramanian, V . Jouault, A. Rastogi, A. Sad’e, A. Jeffares, A. Q. Jiang, A. Cahill, A. Gavaudan, A. Sablayrolles, A. H’eliou, A. You, A. Ehrenberg, A. D. Lo, A. Eliseev, A. Calvi, A. Sooriyarachchi, B. Bout, B. Rozière, B. D. Monicault, C. Lanfranchi, C. Barreau, C. Courtot, D. Grattarola, D. Dabert, D. de Las Casas, E. Chane-Sa...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  56. [58]

    X. Liu, M. Khalifa, and L. Wang. Litcab: Lightweight language model calibration over short- and long-form responses. InThe Twelfth International Conference on Learning Representations,

  57. [59]

    URLhttps://openreview.net/forum?id=jH67LHVOIO

  58. [60]

    X. Liu, T. Chen, L. Da, C. Chen, Z. Lin, and H. Wei. Uncertainty quantification and confidence calibration in large language models: A survey. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V .2, KDD ’25, page 6107–6117, New 13 York, NY , USA, 2025. Association for Computing Machinery. ISBN 9798400714542. doi: 10.11...

    work page doi:10.1145/3711896.3736569 2025

  59. [61]

    Macêdo, T

    D. Macêdo, T. I. Ren, C. Zanchettin, A. L. I. Oliveira, and T. Ludermir. Entropic out-of- distribution detection: Seamless detection of unknown examples.IEEE Transactions on Neural Networks and Learning Systems, 33(6):2350–2364, 2022. doi: 10.1109/TNNLS.2021.3112897

    work page doi:10.1109/tnnls.2021.3112897 2022

  60. [62]

    Malinin and M

    A. Malinin and M. Gales. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations, 2021. URL https://openreview. net/forum?id=jN5y-zb5Q7m

    2021

  61. [63]

    McLaren, L

    M. McLaren, L. Ferrer, D. Castan, and A. Lawson. The speakers in the wild (SITW) speaker recognition database. InProc. Interspeech, San Francisco, Sept. 2016

    2016

  62. [64]

    S. J. Mielke, A. Szlam, E. Dinan, and Y .-L. Boureau. Reducing conversational agents’ over- confidence through linguistic calibration.Transactions of the Association for Computational Linguistics, 10:857–872, 2022. doi: 10.1162/tacl_a_00494. URL https://aclanthology. org/2022.tacl-1.50/

    work page doi:10.1162/tacl_a_00494 2022

  63. [65]

    Morrison, C

    G. Morrison, C. Zhang, and E. Enzinger et. al. Forensic database of voice recordings of 500+ australian english speakers.http://databases.forensic-voice-comparison.net, 2015

    2015

  64. [66]

    G. S. Morrison, P. Rose, and C. Zhang. Protocol for the collection of databases of recordings for forensic-voice-comparison research and practice.Australian Journal of Forensic Sciences, 44(2):155–167, June 2012

    2012

  65. [67]

    M. S. A. Nadeem, J.-D. Zucker, and B. Hanczar. Accuracy-rejection curves (arcs) for comparing classification methods with a reject option. In S. Džeroski, P. Guerts, and J. Rousu, editors, Proceedings of the third International Workshop on Machine Learning in Systems Biology, volume 8 ofProceedings of Machine Learning Research, pages 65–81, Ljubljana, Slo...

    2009

  66. [68]

    In: 2024 IEEE International Symposium on Biomedical Imaging (ISBI), pp

    J. Naushad and I. V oiculescu. Super-trustscore: Reliable failure detection for automated skin lesion diagnosis. In2024 IEEE International Symposium on Biomedical Imaging (ISBI), pages 1–4, 2024. doi: 10.1109/ISBI56570.2024.10635815

    work page doi:10.1109/isbi56570.2024.10635815 2024

  67. [69]

    Peterson.An Introduction to Decision Theory

    M. Peterson.An Introduction to Decision Theory. Cambridge Introductions to Philosophy. Cambridge University Press, 2 edition, 2017

    2017

  68. [70]

    M. M. H. Raiffa. Decision analysis. introductory lectures on choices under uncertainty. Recherches économiques de Louvain, 36(5):527–528, 1970

    1970

  69. [71]

    Russell and P

    S. Russell and P. Norvig.Artificial Intelligence: A Modern Approach. Prentice Hall, 2010

    2010

  70. [72]

    Russell and P

    S. Russell and P. Norvig.Artificial Intelligence: A Modern Approach. Always learning. Pearson, 2016. ISBN 9781292153964. URL https://books.google.com.ar/books?id= XS9CjwEACAAJ

    2016

  71. [73]

    L. J. Savage. The foundations of statistics reconsidered. InProceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics, volume 4, pages 575–587. University of California Press, 1961

    1961

  72. [74]

    L. J. Savage.The foundations of statistics. Courier Corporation, 1972

    1972

  73. [75]

    Socher, A

    R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. InProceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Washington, USA, Oct. 2013

    2013

  74. [76]

    Stengel-Eskin and B

    E. Stengel-Eskin and B. Van Durme. Calibrated interpretation: Confidence estimation in semantic parsing.Transactions of the Association for Computational Linguistics, 11:1213–1231,

  75. [77]

    URLhttps://aclanthology.org/2023.tacl-1.69/

    doi: 10.1162/tacl_a_00598. URLhttps://aclanthology.org/2023.tacl-1.69/

    work page doi:10.1162/tacl_a_00598 2023

  76. [78]

    Q. Team. Qwen3.5: Accelerating productivity with native multimodal agents, February 2026. URLhttps://qwen.ai/blog?id=qwen3.5

    2026

  77. [79]

    V . Team, W. Hong, W. Yu, X. Gu, G. Wang, G. Gan, H. Tang, J. Cheng, J. Qi, J. Ji, L. Pan, S. Duan, W. Wang, Y . Wang, Y . Cheng, Z. He, Z. Su, Z. Yang, Z. Pan, A. Zeng, B. Wang, B. Chen, B. Shi, C. Pang, C. Zhang, D. Yin, F. Yang, G. Chen, J. Xu, J. Zhu, J. Chen, J. Chen, J. Chen, J. Lin, J. Wang, J. Chen, L. Lei, L. Gong, L. Pan, M. Liu, M. Xu, M. Zhang...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  78. [80]

    K. Tian, E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, and C. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In H. Bouamor, J. Pino, and K. Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pag...

    work page doi:10.18653/v1/2023 2023

  79. [81]

    D. Tran, J. Z. Liu, M. W. Dusenberry, D. Phan, M. Collier, J. Ren, K. Han, Z. Wang, Z. E. Mariet, H. Hu, N. Band, T. G. J. Rudner, Z. Nado, J. van Amersfoort, A. Kirsch, R. Jenatton, N. Thain, E. K. Buchanan, K. P. Murphy, D. Sculley, Y . Gal, Z. Ghahramani, J. Snoek, and B. Lakshminarayanan. Plex: Towards reliability using pretrained large model extensio...

    2022

  80. [82]

    Traub, T

    J. Traub, T. J. Bungert, C. T. Lüth, M. Baumgartner, K. Maier-Hein, L. Maier-hein, and P. F. Jaeger. Overcoming Common Flaws in the Evaluation of Selective Classification Systems. Nov

Showing first 80 references.