pith. sign in

arxiv: 2605.01733 · v3 · pith:GQ3K7COZnew · submitted 2026-05-03 · 💻 cs.CV · cs.AI

GEASS: Gated Evidence-Adaptive Selective Caption Trust for Vision-Language Models

Zeshang Li , Shuoyang Zhang , Jiashen Ding This is my paper

Pith reviewed 2026-07-01 00:29 UTC · model grok-4.3

classification 💻 cs.CV cs.AI
keywords vision language modelshallucinationscaption evidencegated decodingselective trustPOPEHallusionBenchdecoder signals
0
0 comments X

The pith

A caption's usefulness for a specific query in vision-language models can be determined from the decoder's own confidence and entropy signals, allowing selective trust to reduce hallucinations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Vision-language models sometimes produce worse answers when given their own generated caption as evidence because the caption pulls attention away from the image. The paper demonstrates this effect varies by query: the same caption helps broad questions but hurts detailed ones depending on whether it covers the asked content. This coverage is detectable from standard decoder outputs like clean-path confidence and entropy reduction without needing attention weights. GEASS uses these to gate and weight the caption's contribution at the logit level in a training-free way. When tested on four models and two benchmarks, it outperforms both standard inference and contrastive decoding with one fixed configuration.

Core claim

The paper establishes that caption utility is per-query and readable from decoder quantities. By building a diagnostic set pairing global and detail questions, it shows the competition for attention is the mechanism, and its sign depends on coverage. GEASS gates the caption by clean path confidence, weights by entropy reduction, and adjusts when pathways disagree, yielding consistent gains across VLMs on POPE and HallusionBench.

What carries the argument

GEASS, a training-free logit-level module that selectively trusts generated captions by gating on decoder confidence and weighting on entropy reduction.

If this is right

  • Per-query gating of evidence improves accuracy over uniform use or no use of captions.
  • Decoder-internal signals suffice to assess evidence utility without external grounding.
  • A single set of thresholds generalizes across different vision-language models.
  • Only two additional forward passes are needed for the improvement.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The attention competition mechanism may apply to other auxiliary inputs like retrieved documents in language models.
  • Fixed thresholds might enable zero-shot deployment on new models if the signals prove robust.
  • This approach could be combined with other hallucination mitigation techniques for further gains.

Load-bearing premise

The decoder's confidence on the clean path and the entropy reduction from adding the caption reliably signal whether the caption covers the queried content, and fixed thresholds transfer to new models.

What would settle it

Running GEASS with the reported fixed thresholds on a fifth VLM or a new hallucination benchmark and finding no improvement or a drop relative to vanilla inference would falsify the claim of reliable transfer.

Figures

Figures reproduced from arXiv: 2605.01733 by Jiashen Ding, Shuoyang Zhang, Zeshang Li.

Figure 1
Figure 1. Figure 1: Caption anchoring effect observed on Qwen2.5-VL-3B with chain-of-thought reasoning. The model’s output (left, in red) closely mirrors the phrasing of the embedded caption (right, in red), demonstrating that captions reshape not only the final answer but the model’s entire reasoning trajectory. methods avoid retraining and intervene at decoding. One line leverages external vision models for post-hoc verific… view at source ↗
Figure 2
Figure 2. Figure 2: Left: Objects contained in the caption generated by InternVL2-8B. Right: Salient objects that are clearly visible in the image but not mentioned in the caption [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Top: With a correct caption mentioning “a dog sitting on the beach,” both Qwen2.5-VL-3B and InternVL2-8B revise their initially incorrect answers from No to Yes, demonstrating confidence grounding. Bottom: With a wrong caption mentioning “a cat sitting on the beach,” both models similarly flip to Yes and fabricate supporting details, demonstrating hallucination amplifica￾tion. The same anchoring mechanism … view at source ↗
Figure 4
Figure 4. Figure 4: Asymmetric per-instance impact of caption errors on Qwen2.5-VL-3B (100 image–question instances): fabrication shifts predictions sharply (∆p = 0.64, 87% flips), while omission is mild on average (∆p = 0.13) but its long tail still flips 11% of answers. ∆p is the caption-induced shift toward the wrong answer; answers flip above the shaded threshold (∆p > 0.4). Inner boxes mark median and IQR; diamonds mark … view at source ↗
Figure 5
Figure 5. Figure 5: Overview of the GEASS pipeline. Given an image I and a question Q, the model first generates a caption C via self-captioning (Stage 1). Two parallel forward passes through the same VLM with shared parameters produce logit vectors zclean (conditioned on I, Q) and zcap (conditioned on I, Q, C) (Stage 2). The adaptive fusion module (Stage 3) computes a confidence gate α that assesses whether the model needs h… view at source ↗
Figure 6
Figure 6. Figure 6: Left: example where caption causes hallucination (need to reduce caption influence). Right: example where caption corrects the prediction (need normal caption steering). 11 [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
read the original abstract

Vision-Language Models (VLMs) hallucinate objects that are not present, and a growing line of work tries to curb this by feeding the model its own generated caption as auxiliary evidence -- assuming that a caption, once available, is something to consume. We show this fails: naively appending a caption can lower accuracy rather than raise it, dropping Qwen2.5-VL-3B$^\dagger$ on HallusionBench by nearly ten points. To understand why, we build \textbf{GD-Probe}, a diagnostic set that pairs a global and a detail question on the same image, so that any difference in caption effect is attributable to the question alone. Caption utility proves to be a \emph{per-query} property: the same caption helps global questions and harms detail ones, through a single mechanism -- an embedded caption competes with the image for attention and pulls the model's evidence onto its own text -- whose sign is set by whether the caption \emph{covers} the queried content. Crucially, this regime is readable from quantities the decoder already emits, with no attention access or grounding. We turn this into \textbf{GEASS} (Gated Evidence-Adaptive Selective Caption Trust), a training-free, logit-level module that decides per query how much of the caption to trust, gating it by the clean path's confidence, weighting it by the entropy reduction it induces, and raising the evidence bar when the two pathways disagree. Across four VLMs and two benchmarks (POPE and HallusionBench), GEASS improves over both vanilla inference and contrastive decoding under a single fixed setting, adding only two forward passes and no parameters.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces GEASS, a training-free logit-level gating module for VLMs that selectively trusts generated captions per query using clean-path confidence and entropy reduction (quantities already emitted by the decoder). It presents GD-Probe, a diagnostic set pairing global and detail questions on the same image, to show that caption utility is a per-query property whose sign depends on whether the caption covers the queried content, and reports that GEASS improves over vanilla inference and contrastive decoding on POPE and HallusionBench across four VLMs under one fixed setting.

Significance. If the results hold, the work provides a practical, parameter-free method to mitigate VLM hallucinations by adaptively gating auxiliary caption evidence, with the diagnostic finding that caption effects are query-dependent and readable from decoder outputs offering a useful insight for reliable inference. The single fixed setting across models and the training-free nature are notable strengths if the transferability is substantiated.

major comments (2)
  1. [Abstract] Abstract and GD-Probe section: the central claim that GEASS improves performance under a single fixed setting across four VLMs rests on thresholds chosen on the diagnostic set, but the manuscript provides no description of the selection procedure, no validation on held-out models, and no evidence that the thresholds are model-agnostic; this directly affects the transferability assertion given differing logit scales across architectures.
  2. [GD-Probe and evaluation] GD-Probe results and evaluation sections: the readability claim (that caption coverage is indicated by clean-path confidence and entropy reduction) and the reported gains lack error bars, ablation studies on threshold sensitivity, and details on how the fixed setting was chosen, leaving the robustness of the per-query mechanism and the cross-model consistency unverified.
minor comments (1)
  1. The abstract and results tables would be clearer with explicit mention of the exact numerical thresholds used in the fixed setting and the number of forward passes required.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. We address each major comment below, clarifying the threshold selection process and committing to strengthen the evaluation sections with additional details and analyses in the revision.

read point-by-point responses

  1. Referee: [Abstract] Abstract and GD-Probe section: the central claim that GEASS improves performance under a single fixed setting across four VLMs rests on thresholds chosen on the diagnostic set, but the manuscript provides no description of the selection procedure, no validation on held-out models, and no evidence that the thresholds are model-agnostic; this directly affects the transferability assertion given differing logit scales across architectures.

    Authors: The thresholds were derived from the GD-Probe diagnostic observations to capture the per-query distinction in caption utility (global vs. detail) while enforcing a single setting that applies uniformly without per-model retuning. We will revise the manuscript to include an explicit description of this selection procedure, including the specific criteria and values used. The four VLMs span different architectures and logit scales, and the gating relies on relative decoder quantities (confidence, entropy reduction, pathway disagreement) rather than absolute scales; however, we acknowledge the absence of held-out model validation and will note this as a limitation while emphasizing the observed consistency across the tested models. revision: yes

  2. Referee: [GD-Probe and evaluation] GD-Probe results and evaluation sections: the readability claim (that caption coverage is indicated by clean-path confidence and entropy reduction) and the reported gains lack error bars, ablation studies on threshold sensitivity, and details on how the fixed setting was chosen, leaving the robustness of the per-query mechanism and the cross-model consistency unverified.

    Authors: We agree that error bars, threshold sensitivity ablations, and expanded details on the fixed-setting choice would strengthen verification of robustness and cross-model consistency. The fixed setting was selected to reflect the query-dependent caption effects identified in GD-Probe and applied uniformly. In the revised manuscript we will add error bars to the POPE and HallusionBench results, include an ablation study on threshold sensitivity, and expand the description of how the fixed setting was determined from the diagnostic set. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper constructs GEASS explicitly from decoder quantities (clean-path confidence, entropy reduction) observed on the separate GD-Probe diagnostic set, then applies fixed thresholds empirically across VLMs on POPE and HallusionBench. No equations or steps reduce by construction to fitted inputs, self-citations, or ansatzes; the gating rule is presented as a direct mapping from already-emitted quantities rather than a renamed prediction or load-bearing self-reference. The derivation chain remains self-contained with independent empirical validation.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no explicit free parameters, axioms, or invented entities are stated. The method relies on quantities already produced by the decoder and a fixed gating rule whose selection criteria are not detailed.

pith-pipeline@v0.9.1-grok · 5839 in / 1199 out tokens · 21968 ms · 2026-07-01T00:29:54.916345+00:00 · methodology

Review history (3 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 10 canonical work pages · 5 internal anchors

  1. [1]

    Qwen2.5-VL Technical Report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling

    Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., and Lin, D. Sharegpt4v: Improving large multi-modal models with better captions. InEuropean Conference on Computer Vision, pp. 370–387. Springer, 2024a. Chen, Z., Wang, W., Cao, Y ., Liu, Y ., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al. Expanding per- formance boundaries o...

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    A., and Luo, J

    Hu, Y ., Hua, H., Yang, Z., Shi, W., Smith, N. A., and Luo, J. Promptcap: Prompt-guided task-aware image captioning. arXiv preprint arXiv:2211.09699,

    work page arXiv

  4. [4]

    and Song, M

    Lee, J. and Song, M. Retrieval visual contrastive decoding to mitigate object hallucinations in large vision-language models. InFindings of the Association for Computational Linguistics: ACL 2025, pp. 8200–8219,

    2025

  5. [5]

    Cai: Caption- sensitive attention intervention for mitigating object hallu- cination in large vision-language models.arXiv preprint arXiv:2506.23590, 2025a

    Li, Q., Ye, Z., Feng, X., Zhong, W., Qin, L., Chen, R., Li, B., Jiang, K., Wang, Y ., Liu, T., et al. Cai: Caption- sensitive attention intervention for mitigating object hallu- cination in large vision-language models.arXiv preprint arXiv:2506.23590, 2025a. Li, Q., Ye, Z., Feng, X., Zhong, W., Qin, L., Chen, R., Huang, L., Li, B., Jiang, K., Wang, Y ., e...

    work page arXiv

  6. [6]

    Evaluating object hallucination in large vision-language models

    Li, Y ., Du, Y ., Zhou, K., Wang, J., Zhao, X., and Wen, J.-R. Evaluating object hallucination in large vision-language models. InProceedings of the 2023 conference on empiri- cal methods in natural language processing, pp. 292–305,

    2023

  7. [7]

    Li, Z., Shi, H., Gao, Y ., Liu, D., Wang, Z., Chen, Y ., Liu, T., Zhao, L., Wang, H., and Metaxas, D. N. The hidden life of tokens: Reducing hallucination of large vision- language models via visual information steering.arXiv preprint arXiv:2502.03628, 2025b. Lin, T.-Y ., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll ´ar, P., and Zitnic...

    work page arXiv

  8. [8]

    Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning

    9 GEASS: Gated Evidence-Adaptive Selective Caption Trust for Vision-Language Models Liu, C., Wang, C., Peng, Y ., and Li, Z. Zvqaf: Zero-shot visual question answering with feedback from large lan- guage models.Neurocomputing, 580:127505, 2024a. Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y ., and Wang, L. Mitigating hallucination in large multi-modal mod...

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Liu, H., Li, C., Li, Y ., and Lee, Y . J. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 26296–26306, 2024b. Park, W., Kim, W., Kim, J., and Do, J. Second: Mitigat- ing perceptual hallucination in vision-language models via selective and contrastive decoding.arXiv...

    work page arXiv

  10. [10]

    A., Burns, K., Darrell, T., and Saenko, K

    Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., and Saenko, K. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pp. 4035–4045,

    2018

  11. [11]

    A., and Kundu, S

    Sarkar, S., Che, Y ., Gavin, A., Beerel, P. A., and Kundu, S. Mitigating hallucinations in vision-language models through image-guided head suppression. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 12492–12511,

    2025

  12. [12]

    Aligning large multi- modal models with factually augmented rlhf

    Sun, Z., Shen, S., Cao, S., Liu, H., Li, C., Shen, Y ., Gan, C., Gui, L., Wang, Y .-X., Yang, Y ., et al. Aligning large multi- modal models with factually augmented rlhf. InFindings of the Association for Computational Linguistics: ACL 2024, pp. 13088–13110,

    2024

  13. [13]

    Mitigating hallucinations in large vision-language models with in- ternal fact-based contrastive decoding.arXiv preprint arXiv:2502.01056,

    Wang, C., Zhou, X., Fu, W., and Zhou, Y . Mitigating hallucinations in large vision-language models with in- ternal fact-based contrastive decoding.arXiv preprint arXiv:2502.01056,

    work page arXiv

  14. [14]

    Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models

    Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y ., Su, W., Shao, J., et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479,

    work page internal anchor Pith review Pith/arXiv arXiv