pith. sign in

arxiv: 2605.02443 · v2 · pith:AFBU3EZFnew · submitted 2026-05-04 · 💻 cs.CL

HalluScan: A Systematic Benchmark for Detecting and Mitigating Hallucinations in Instruction-Following LLMs

Ahmed Cherif This is my paper

Pith reviewed 2026-05-25 06:47 UTC · model grok-4.3

classification 💻 cs.CL
keywords hallucination detectionlarge language modelsbenchmarkNLI verificationadaptive routinginstruction followingerror analysismitigation
0
0 comments X

The pith

NLI Verification achieves the highest AUROC of 0.88 for detecting hallucinations across 72 LLM configurations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces HalluScan as a benchmark to compare hallucination detection methods in instruction-following large language models. It evaluates six methods on four model families and three domains while introducing HalluScore as a composite metric and Adaptive Detection Routing as a cost-saving algorithm. The central finding is that NLI Verification outperforms the other methods tested. A sympathetic reader would care because consistent detection of unfaithful or incorrect outputs could make LLMs more reliable for downstream tasks. The work also breaks down error cascades to show how hallucination types differ by domain.

Core claim

The authors present HalluScan as a benchmark that evaluates hallucination detection and mitigation in 72 configurations. They report that NLI Verification reaches an AUROC of 0.88, the highest among methods tested, while RAV reaches 0.66. They introduce HalluScore which correlates at 0.41 with human judgments and Adaptive Detection Routing which reduces cost by a factor of 2.0 with only 0.1 percent AUROC degradation. The benchmark also reveals variation in hallucination error types across domains through systematic error cascade decomposition.

What carries the argument

HalluScan, the benchmark framework that systematically varies six detection methods, four model families, and three domains across 72 configurations to measure AUROC and human correlation.

Load-bearing premise

The 3 domains, 4 model families, and 6 detection methods chosen for the 72 configurations are representative enough that the reported AUROCs and correlations generalize beyond the specific test cases used.

What would settle it

Evaluating the same six detection methods on a new domain or model family outside the tested set and finding that NLI Verification's AUROC falls below 0.8 or changes ranking would falsify the generalizability of the results.

read the original abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks, yet they remain susceptible to hallucinations -- generating content that is factually incorrect, unfaithful to provided context, or misaligned with user instructions. We present HalluScan, a comprehensive benchmark framework that systematically evaluates hallucination detection and mitigation across 72 configurations spanning 6 detection methods, 4 open-weight model families, and 3 diverse domains. We introduce three key contributions: (1) HalluScore, a novel composite metric that achieves a Pearson correlation of r = 0.41 with human expert judgments; (2) Adaptive Detection Routing (ADR), an intelligent routing algorithm achieving 2.0x cost reduction with only 0.1% AUROC degradation; and (3) systematic error cascade decomposition revealing substantial variation in hallucination error types across domains. Our experiments reveal that NLI Verification achieves the highest overall AUROC of 0.88, while RAV achieves the second-highest AUROC of 0.66.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper presents HalluScan, a benchmark evaluating hallucination detection and mitigation in instruction-following LLMs across 72 configurations (6 detection methods, 4 open-weight model families, 3 domains). It introduces HalluScore (Pearson r = 0.41 with human experts), Adaptive Detection Routing (ADR) achieving 2.0x cost reduction with 0.1% AUROC drop, and error cascade decomposition. Experiments conclude that NLI Verification attains the highest overall AUROC of 0.88 while RAV reaches 0.66.

Significance. If the empirical results hold under broader sampling, the work supplies a useful head-to-head comparison of detection techniques and a practical routing algorithm that trades negligible accuracy for substantial cost savings. The composite HalluScore and domain-wise error breakdown are constructive additions. The limited scope (only three domains and four families) nevertheless caps the strength of any claim that NLI Verification is generally superior.

major comments (2)
  1. [Abstract and §4] Abstract and §4 (Experiments): The headline AUROC figures (NLI Verification 0.88, RAV 0.66) and the superiority claim rest on 72 configurations drawn from only three domains and four model families. No selection criteria, diversity statistics, or sensitivity analysis for these choices are supplied, which directly undermines generalization of the reported performance ordering.
  2. [§3] §3 (Benchmark Construction): The error-cascade decomposition is presented as revealing substantial cross-domain variation, yet the manuscript gives no justification that the chosen three domains adequately sample instruction-following hallucination types (long-context inconsistency, multi-turn drift, domain-specific factual drift). This choice is load-bearing for the decomposition's interpretive value.
minor comments (1)
  1. [Abstract] Abstract: The specific numerical claims (AUROCs, r = 0.41, 2.0x cost reduction) are stated without cross-reference to the sections that detail human-judgment collection protocol and pre-specification of the 72 configurations.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below and outline targeted revisions to qualify scope and add justification where feasible.

read point-by-point responses

  1. Referee: [Abstract and §4] Abstract and §4 (Experiments): The headline AUROC figures (NLI Verification 0.88, RAV 0.66) and the superiority claim rest on 72 configurations drawn from only three domains and four model families. No selection criteria, diversity statistics, or sensitivity analysis for these choices are supplied, which directly undermines generalization of the reported performance ordering.

    Authors: We agree that the limited scope (three domains, four families) and absence of sensitivity analysis constrain generalization claims. We will revise the abstract and §4 to state that results hold within the evaluated configurations, add brief selection rationale for domains and models, and insert a limitations paragraph noting the need for broader sampling. This is a partial revision focused on text qualification rather than new experiments. revision: partial

  2. Referee: [§3] §3 (Benchmark Construction): The error-cascade decomposition is presented as revealing substantial cross-domain variation, yet the manuscript gives no justification that the chosen three domains adequately sample instruction-following hallucination types (long-context inconsistency, multi-turn drift, domain-specific factual drift). This choice is load-bearing for the decomposition's interpretive value.

    Authors: We concur that explicit justification is needed for the domains' coverage of hallucination types. In revision we will expand §3 with a short rationale linking each domain to relevant hallucination characteristics drawn from prior work, while clarifying that the decomposition is exploratory rather than exhaustive. This constitutes a partial revision. revision: partial

Circularity Check

0 steps flagged

No circularity: empirical benchmark with direct measurements only

full rationale

The paper reports results from running 6 detection methods on 4 models across 3 domains (72 configurations total). All headline numbers (NLI AUROC 0.88, RAV AUROC 0.66, HalluScore Pearson r=0.41, ADR 2.0x cost reduction) are direct empirical outputs of those runs. No equations, derivations, fitted parameters renamed as predictions, or self-citation chains appear in the abstract or described contributions. The study is self-contained; its claims rest on the experimental data collected rather than on any reduction to prior inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters, axioms, or invented entities are described; the work relies on standard machine-learning evaluation practices such as AUROC and Pearson correlation.

pith-pipeline@v0.9.0 · 5711 in / 1147 out tokens · 28093 ms · 2026-05-25T06:47:27.335808+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 56 canonical work pages · 14 internal anchors

  1. [1]

    Qwen Technical Report

    Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al.: Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  2. [2]

    SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model

    Allal, L.B., Lozhkov, A., Penedo, G., Wolf, T., Werra, L.: Smollm2: When smol goes big – data-centric training of a small language model. arXiv preprint arXiv:2502.02737 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  3. [3]

    Advances in Neural Information Processing Systems33, 1877–1901 (2020)

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A.,et al.: Language models are few-shot learners. Advances in Neural Information Processing Systems33, 1877–1901 (2020)

    work page 1901

  4. [4]

    arXiv preprint arXiv:2402.17834 (2024) 34

    Bellagente, M., Tow, J., Mahan, D., Phung, D., Zhuravinskyi, M., Knighton, R., et al.: Stable lm 2 1.6b technical report. arXiv preprint arXiv:2402.17834 (2024) 34

    work page arXiv 2024

  5. [5]

    ACM Computing Surveys55(12), 1–38 (2023)

    Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y.J., Madotto, A., Fung, P.: Survey of hallucination in natural language generation. ACM Computing Surveys55(12), 1–38 (2023)

    work page 2023

  6. [6]

    Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models

    Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al.: Siren’s song in the AI ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.01219 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  7. [7]

    In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp

    Lin, S., Hilton, J., Evans, O.: TruthfulQA: Measuring how models mimic human falsehoods. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp. 3214–3252 (2022)

    work page 2022

  8. [8]

    Transactions of the Association for Computational Linguistics7, 453–466 (2019)

    Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K.: Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics7, 453–466 (2019)

    work page 2019

  9. [9]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., Tafjord, O.: Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457 (2018)

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  10. [10]

    HaluEval: A large-scale hallucination evaluation benchmark for large language models.arXiv preprint arXiv:2305.11747, 2023

    Li, J., Cheng, X., Zhao, W.X., Nie, J.-Y., Wen, J.-R.: HaluEval: A large-scale hallucination evaluation benchmark for large language models. arXiv preprint arXiv:2305.11747 (2023)

    work page arXiv 2023

  11. [11]

    arXiv preprint arXiv:2310.00657 (2023)

    Chen, W., Ma, M., Guo, J., Wang, B.: FELM: Benchmarking faithfulness evaluation of large language models. arXiv preprint arXiv:2310.00657 (2023)

    work page arXiv 2023

  12. [12]

    arXiv preprint arXiv:2501.03916 (2025)

    Wu, Z., Zhang, H., Chen, W.: HalluLens: A multi-dimensional benchmark for hal- lucination evaluation in large language models. arXiv preprint arXiv:2501.03916 (2025)

    work page arXiv 2025

  13. [13]

    FActScore: Fine-grained atomic evaluation of factual precision in long form text generation

    Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-t., Koh, P.W., Iyyer, M., Zettle- moyer, L., Hajishirzi, H.: FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251 (2023)

    work page arXiv 2023

  14. [14]

    arXiv preprint arXiv:2502.01234 (2025)

    Park, J., Kim, S., Lee, J.: PHANTOM: A benchmark for probing hallucinations with adversarial and natural transformations on model outputs. arXiv preprint arXiv:2502.01234 (2025)

    work page arXiv 2025

  15. [15]

    A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions

    Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al.: A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  16. [16]

    In: Advances in Neural Information Processing Systems (2020)

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K¨ uttler, H., Lewis, M., Yih, W.-t., Rockt¨ aschel, T.,et al.: Retrieval-augmented generation for 35 knowledge-intensive NLP tasks. In: Advances in Neural Information Processing Systems (2020)

    work page 2020

  17. [17]

    arXiv preprint arXiv:2210.08726 , year=

    Gao, L., Dai, Z., Pasupat, P., Chen, A., Chaganty, A.T., Fan, Y., Zhao, V.Y., Lao, N., Lee, H., Juan, D.-C., et al.: RARR: Researching and revising what language models say, using language models. arXiv preprint arXiv:2210.08726 (2023)

    work page arXiv 2023

  18. [18]

    In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp

    Maynez, J., Narayan, S., Bohnet, B., McDonald, R.: On faithfulness and factu- ality in abstractive summarization. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1906–1919 (2020)

    work page 1906

  19. [19]

    arXiv preprint arXiv:1910.12840 (2019)

    Kry´ sci´ nski, W., McCann, B., Xiong, C., Socher, R.: Evaluating the factual consistency of abstractive text summarization. arXiv preprint arXiv:1910.12840 (2020)

    work page arXiv 1910

  20. [20]

    TRUE: Re-evaluating factual consistency evaluation.arXiv preprint arXiv:2204.04991, 2022

    Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kuber, D., Choshen, L., Sznajder, B., Toledo, N., Schiller, D., Slonim, N.: TRUE: Re-evaluating factual consistency evaluation. arXiv preprint arXiv:2204.04991 (2022)

    work page arXiv 2022

  21. [21]

    In: Transactions of the Association for Computational Linguistics, vol

    Laban, P., Schnabel, T., Bennett, P.N., Hearst, M.A.: SummaC: Re-visiting NLI- based models for inconsistency detection in summarization. In: Transactions of the Association for Computational Linguistics, vol. 10, pp. 163–177 (2022)

    work page 2022

  22. [22]

    Instruction-Following Evaluation for Large Language Models

    Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., Hou, L.: Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  23. [23]

    In: Proceedings of the 11th International Conference on Learning Representations (2023)

    Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D.: Self-consistency improves chain of thought reasoning in lan- guage models. In: Proceedings of the 11th International Conference on Learning Representations (2023)

    work page 2023

  24. [24]

    SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models

    Manakul, P., Liusie, A., Gales, M.J.F.: SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  25. [25]

    Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

    Kuhn, L., Gal, Y., Farquhar, S.: Semantic uncertainty: Linguistic invari- ances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  26. [26]

    Alignscore: Evaluating factual consistency with a unified alignment function,

    Zha, Y., Yang, Y., Li, R., Hu, Z.: AlignScore: Evaluating factual consistency with a unified alignment function. arXiv preprint arXiv:2305.16739 (2023)

    work page arXiv 2023

  27. [27]

    In: Advances in Neural Information Processing Systems (2023) 36

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E.P.,et al.: Judging LLM-as-a-judge with MT-Bench and chatbot arena. In: Advances in Neural Information Processing Systems (2023) 36

    work page 2023

  28. [28]

    Blog post (2023)

    Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., et al.: Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. Blog post (2023)

    work page 2023

  29. [29]

    arXiv preprint arXiv:2405.01535 , year=

    Kim, S., Shin, J., Cho, Y., Jang, J., Longpre, S., Lee, H., Yun, S., Shin, S., Kim, S., Thorne, J., et al.: Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01535 (2024)

    work page arXiv 2024

  30. [30]

    Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection

    Asai, A., Wu, Z., Wang, Y., Sil, A., Hajishirzi, H.: Self-RAG: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  31. [31]

    Advances in Neural Information Processing Systems35, 27730–27744 (2022)

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A.,et al.: Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems35, 27730–27744 (2022)

    work page 2022

  32. [32]

    Towards Understanding Sycophancy in Language Models

    Perez, E., Ringer, S., Lukoˇ si¯ ut˙ e, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al.: Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  33. [33]

    Advances in Neural Information Processing Systems (2023)

    Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al.: Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems (2023)

    work page 2023

  34. [34]

    Large Language Models Cannot Self-Correct Reasoning Yet

    Huang, J., Gu, S.-H., Hosseini, H., Zhao, J., Rao, K., Chen, D.: Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  35. [35]

    Advances in Neural Information Processing Systems (2023)

    Li, K., Patel, O., Vi´ egas, F., Pfister, H., Wattenberg, M.: Inference-time inter- vention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems (2023)

    work page 2023

  36. [36]

    arXiv preprint arXiv:2305.14739 , year =

    Shi, W., Han, X., Lewis, M., Tsvetkov, Y., Zettlemoyer, L., Yih, W.-t.: Trust- ing your evidence: Hallucinate less with context-aware decoding. arXiv preprint arXiv:2305.14739 (2023)

    work page arXiv 2023

  37. [37]

    Chain-of-Verification Reduces Hallucination in Large Language Models

    Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., Weston, J.: Chain-of-verification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  38. [38]

    A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of

    Varshney, N., Yao, W., Zhang, H., Chen, J., Yu, D.: A stitch in time saves nine: Detecting and mitigating hallucinations of LLMs by validating low-confidence generation. arXiv preprint arXiv:2307.03987 (2023)

    work page arXiv 2023

  39. [39]

    arXiv preprint arXiv:2305.15852 (2024) 37

    M¨ undler, N., He, J., Jenko, S., Vechev, M.: Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation. arXiv preprint arXiv:2305.15852 (2024) 37

    work page arXiv 2024

  40. [40]

    The Internal State of an LLM Knows When It's Lying

    Azaria, A., Mitchell, T.: The internal state of an LLM knows when it’s lying. arXiv preprint arXiv:2304.13734 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  41. [41]

    arXiv preprint arXiv:2402.03744 , year=

    Chen, C., Ding, K., Liu, K., Ji, X., Chen, E.: Inside: LLMs’ internal states retain the power of hallucination detection. arXiv preprint arXiv:2402.03744 (2024)

    work page arXiv 2024

  42. [42]

    arXiv preprint arXiv:2407.07071 (2024)

    Chuang, Y.-S., Xie, L., Li, Y.-N., Hemphill, R., Sadigh, D., Kim, Y.: Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps. arXiv preprint arXiv:2407.07071 (2024)

    work page arXiv 2024

  43. [43]

    Mishra, A

    Mishra, A., Celikyilmaz, A., Hakkani-Tur, D.: Fine-grained hallucination detec- tion and editing for language models. arXiv preprint arXiv:2401.06855 (2024)

    work page arXiv 2024

  44. [44]

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A

    Tang, L., Upadhyay, S., Pagnoni, G., Rahimi, A., Celikyilmaz, A., Petroni, F.: MiniCheck: Efficient fact-checking of LLMs on grounding documents. arXiv preprint arXiv:2404.10774 (2024)

    work page arXiv 2024

  45. [45]

    arXiv preprint arXiv:2305.06311 (2023)

    Yue, X., Wang, B., Chen, Z., Zhang, K., Su, Y., Sun, H.: Automatic evaluation of attribution by large language models. arXiv preprint arXiv:2305.06311 (2023)

    work page arXiv 2023

  46. [46]

    arXiv preprint arXiv:2310.08951 (2023)

    Lei, D., Li, Y., Hu, M., Wang, M., Yun, V., Ching, E., Kamal, E.: Chain of natural language inference for reducing large language model ungrounded hallucinations. arXiv preprint arXiv:2310.08951 (2023)

    work page arXiv 2023

  47. [47]

    arXiv preprint arXiv:2407.08039 (2024)

    Zhang, Y., Li, S., Liu, J., He, P., Ji, H.: Knowledge overshadowing causes amal- gamated hallucination in large language models. arXiv preprint arXiv:2407.08039 (2024)

    work page arXiv 2024

  48. [48]

    arXiv preprint arXiv:2403.06872 (2024)

    Sun, Y., Xiao, Z., Yin, Y., Wei, J., Huang, S., Li, L.: Benchmarking hallucina- tion in large language models based on unanswerable math word problem. arXiv preprint arXiv:2403.06872 (2024)

    work page arXiv 2024

  49. [49]

    Language Models (Mostly) Know What They Know

    Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al.: Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221 (2022)

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  50. [50]

    In: Proceedings of the 9th International Conference on Learning Representations (2021)

    He, P., Liu, X., Gao, J., Chen, W.: DeBERTa: Decoding-enhanced BERT with disentangled attention. In: Proceedings of the 9th International Conference on Learning Representations (2021)

    work page 2021

  51. [51]

    Transactions on Machine Learning Research (2022)

    Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., Grave, E.: Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research (2022)

    work page 2022

  52. [52]

    In: Proceedings of the 2020 Conference on Empirical 38 Methods in Natural Language Processing: System Demonstrations, pp

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M.,et al.: Transformers: State-of-the-art natu- ral language processing. In: Proceedings of the 2020 Conference on Empirical 38 Methods in Natural Language Processing: System Demonstrations, pp. 38–45 (2020)

    work page 2020

  53. [53]

    Journal of Machine Learning Research12, 2825–2830 (2011)

    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V.,et al.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research12, 2825–2830 (2011)

    work page 2011

  54. [54]

    Biometrics Bulletin 1(6), 80–83 (1945)

    Wilcoxon, F.: Individual comparisons by ranking methods. Biometrics Bulletin 1(6), 80–83 (1945)

    work page 1945

  55. [55]

    Lawrence Erlbaum Associates, ??? (1988)

    Cohen, J.: Statistical Power Analysis for the Behavioral Sciences, 2nd edn. Lawrence Erlbaum Associates, ??? (1988)

    work page 1988

  56. [56]

    Chapman & Hall/CRC, ??? (1993) 39

    Efron, B., Tibshirani, R.J.: An Introduction to the Bootstrap. Chapman & Hall/CRC, ??? (1993) 39

    work page 1993