pith. sign in

arxiv: 2305.10355 · v3 · pith:555TZ7WGnew · submitted 2023-05-17 · 💻 cs.CV · cs.CL· cs.MM

Evaluating Object Hallucination in Large Vision-Language Models

Yifan Li , Yifan Du , Kun Zhou , Jinpeng Wang , Wayne Xin Zhao , Ji-Rong Wen This is my paper

Pith reviewed 2026-05-11 13:38 UTC · model grok-4.3

classification 💻 cs.CV cs.CLcs.MM
keywords object hallucinationlarge vision-language modelsevaluation methodPOPEvisual instructionsmultimodal generationimage captioning
0
0 comments X

The pith

Large vision-language models often describe objects absent from the given image, especially those frequent in instructions or co-occurring with visible items.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines object hallucination in large vision-language models created by pairing strong language models with vision encoders. Experiments across representative models reveal that these systems generate image descriptions containing objects not present in the actual images. The authors observe that visual instructions bias the output toward objects that appear often in training data or alongside objects that are truly visible. Existing evaluation approaches vary with input phrasing and output style, so the paper introduces POPE, a polling-based query technique that measures hallucination more consistently across models.

Core claim

Large vision-language models suffer from severe object hallucination by generating objects inconsistent with the target images. Objects that frequently occur in the visual instructions or co-occur with the image objects are obviously prone to be hallucinated. Existing evaluation methods might be affected by the input instructions and generation styles of LVLMs, therefore a polling-based query method called POPE evaluates the object hallucination in a more stable and flexible way.

What carries the argument

POPE, a polling-based query method that asks the model yes/no questions about the presence of candidate objects in a fixed polling format to measure hallucination rates.

If this is right

  • Visual instructions should be designed to minimize exposure to frequent or co-occurring objects to lower hallucination rates.
  • POPE allows consistent ranking of different LVLMs on hallucination without dependence on their particular generation styles.
  • Models will continue to favor hallucinated objects that match patterns in their training instructions unless those patterns are altered.
  • Improved evaluation reveals specific objects most likely to be invented, guiding targeted fixes in training data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Hallucination may arise when language-model priors about common object co-occurrences override the actual visual signal.
  • POPE could be extended to probe other hallucination types such as attributes or relations beyond objects.
  • Widespread use of POPE on new models would let researchers track whether scaling or new training techniques actually reduce the problem.
  • If polling reveals systematic over-generation of certain object classes, retraining with balanced negative examples might help.

Load-bearing premise

The selected representative LVLMs and visual instruction datasets are sufficiently typical of the broader class of models, and the polling queries in POPE do not introduce new systematic biases in measuring hallucination.

What would settle it

Run POPE and prior evaluation methods on the same set of model outputs, then compare both against human judgments of object presence in the images; if POPE scores remain stable while prior scores shift with instruction wording, the claim holds.

read the original abstract

Inspired by the superior language abilities of large language models (LLM), large vision-language models (LVLM) have been recently explored by integrating powerful LLMs for improving the performance on complex multimodal tasks. Despite the promising progress on LVLMs, we find that LVLMs suffer from the hallucination problem, i.e. they tend to generate objects that are inconsistent with the target images in the descriptions. To investigate it, this work presents the first systematic study on object hallucination of LVLMs. We conduct the evaluation experiments on several representative LVLMs, and show that they mostly suffer from severe object hallucination issue. We further discuss that the visual instructions may influence the hallucination, and find that: objects that frequently occur in the visual instructions or co-occur with the image objects, are obviously prone to be hallucinated by LVLMs. Besides, we find that existing evaluation methods might be affected by the input instructions and generation styles of LVLMs. Thus, we further design an improved evaluation method for object hallucination by proposing a polling-based query method called POPE. Experiment results demonstrate that our POPE can evaluate the object hallucination in a more stable and flexible way. Our codes and data are publicly available at https://github.com/RUCAIBox/POPE.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 3 minor

Summary. The paper conducts the first systematic study of object hallucination in large vision-language models (LVLMs). Experiments on representative LVLMs show severe hallucination, with objects frequent in visual instructions or co-occurring with image objects being especially prone to generation. Existing evaluation methods are critiqued for sensitivity to instructions and generation style, leading to the proposal of POPE, a polling-based yes/no query method claimed to offer more stable and flexible evaluation. Code and data are released publicly.

Significance. If the central empirical patterns and POPE evaluation hold, the work is significant for multimodal AI research: it quantifies a reliability issue in LVLMs that affects downstream tasks such as captioning and VQA, identifies actionable instruction-related biases, and supplies a practical polling protocol plus public resources for reproducible benchmarking. The explicit release of code and data is a clear strength that supports follow-on mitigation studies.

major comments (3)
  1. [§3] §3 (Experiments): The manuscript does not report exact sample sizes (number of images and queries per model/dataset), controls for prompt variation, or statistical tests (e.g., confidence intervals or significance tests) used to establish the 'severe' hallucination rates and frequency/co-occurrence patterns; without these, the quantitative claims cannot be fully verified.
  2. [§4] §4 (POPE): The claim that POPE evaluates the same underlying hallucination phenomenon as the free-form generation experiments rests on an untested premise; no side-by-side correlation analysis or ablation is presented showing that yes/no polling rates reproduce the frequency and co-occurrence effects observed in open-ended outputs, raising the possibility that POPE instead measures query compliance or yes-bias.
  3. [§4.2] §4.2 (Comparison to prior methods): The superiority of POPE over existing metrics is asserted via stability and flexibility, yet the paper provides no quantitative metric (e.g., variance across prompt styles or inter-rater agreement) demonstrating reduced sensitivity; this is load-bearing for the central methodological contribution.
minor comments (3)
  1. [Abstract] Abstract: Key quantitative findings (e.g., hallucination percentages per model) are omitted; adding one or two headline numbers would improve clarity.
  2. Figure captions and tables: Ensure all axes and legends explicitly label hallucination rate versus object frequency or co-occurrence to avoid ambiguity in interpreting the reported patterns.
  3. Notation: Define 'visual instructions' and 'co-occurrence' operationally in the main text on first use, as these terms are central to the claimed patterns.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed comments, which help clarify the presentation of our empirical findings and the justification for POPE. We address each major comment below and commit to revisions that strengthen the quantitative rigor and validation of our claims without altering the core contributions.

read point-by-point responses

  1. Referee: [§3] §3 (Experiments): The manuscript does not report exact sample sizes (number of images and queries per model/dataset), controls for prompt variation, or statistical tests (e.g., confidence intervals or significance tests) used to establish the 'severe' hallucination rates and frequency/co-occurrence patterns; without these, the quantitative claims cannot be fully verified.

    Authors: We agree that explicit reporting of sample sizes, prompt controls, and uncertainty estimates is necessary for verifiability. The experiments used the full COCO val2014 set (approximately 40k images) for the primary frequency and co-occurrence analyses across models, with 5k-image subsets for efficiency in some LVLM evaluations and 1k random samples for instruction-variation ablations; all queries per image were fixed to the same template to control prompt variation. In the revision we will add a dedicated table listing exact image counts and query counts per model and dataset, state that a single fixed prompt template was used across all models for the main results, and report bootstrap 95% confidence intervals on the reported hallucination percentages and co-occurrence correlations to substantiate the severity claims. revision: yes

  2. Referee: [§4] §4 (POPE): The claim that POPE evaluates the same underlying hallucination phenomenon as the free-form generation experiments rests on an untested premise; no side-by-side correlation analysis or ablation is presented showing that yes/no polling rates reproduce the frequency and co-occurrence effects observed in open-ended outputs, raising the possibility that POPE instead measures query compliance or yes-bias.

    Authors: The design of POPE deliberately decouples object hallucination measurement from open-ended generation style by using balanced yes/no queries, but we acknowledge that we did not present a direct quantitative link showing that the same frequency and co-occurrence biases appear under POPE. In the revised manuscript we will add a new analysis that computes per-object hallucination rates under both free-form captioning and POPE on the same set of images and models, reports Pearson correlations between the two, and includes an ablation that balances positive/negative object queries to quantify any yes-bias. This will either confirm that POPE reproduces the key patterns or allow us to clarify the precise relationship between the two evaluation regimes. revision: yes

  3. Referee: [§4.2] §4.2 (Comparison to prior methods): The superiority of POPE over existing metrics is asserted via stability and flexibility, yet the paper provides no quantitative metric (e.g., variance across prompt styles or inter-rater agreement) demonstrating reduced sensitivity; this is load-bearing for the central methodological contribution.

    Authors: We presented qualitative evidence that prior metrics vary with instruction phrasing and generation length while POPE remains consistent, but we did not supply a quantitative stability metric such as variance across prompt variants. We will revise §4.2 to include a controlled ablation that applies five distinct prompt phrasings to the same images and models, computes the standard deviation of hallucination rates for POPE versus CHAIR and other baselines, and reports these variance numbers together with the original stability claims. This will provide the requested quantitative support for reduced sensitivity. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical evaluation is self-contained

full rationale

The paper performs direct empirical evaluations of object hallucination on representative LVLMs using public datasets and existing methods, then proposes POPE as a polling-based alternative. No derivations, fitted parameters, or predictions reduce by construction to the paper's own inputs or self-citations. Claims about hallucination severity, frequency effects, and POPE's stability rest on experimental comparisons rather than self-referential definitions or load-bearing self-citations. The work is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claims rest on the domain assumption that object hallucination can be reliably detected via object presence checks and that the tested models and instructions represent the general behavior of LVLMs.

axioms (1)
  • domain assumption Object hallucination is defined as generating objects inconsistent with the target images in the descriptions.
    This definition underpins all evaluation experiments and the design of POPE.

pith-pipeline@v0.9.0 · 5546 in / 1208 out tokens · 52159 ms · 2026-05-11T13:38:09.947713+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

    cs.CL 2023-11 unverdicted novelty 8.0

    MMMU provides 11.5K heterogeneous college-level multimodal questions that current models solve at 56-59% accuracy, establishing a new standard for expert multimodal evaluation.

  2. MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs

    cs.LG 2026-06 unverdicted novelty 7.0

    MODE decomposes expert selection frequency by modality, filters redundant vision tokens, adds per-modality sensitivity, and uses ILP to assign bit-widths, limiting average loss to 2.9% at W3A16 on MoE-MLLMs.

  3. P$^2$-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization

    cs.CV 2026-06 unverdicted novelty 7.0

    P²-DPO generates on-policy preference pairs targeting focus-and-enhance perception and visual robustness, combined with a calibration loss, to reduce hallucinations in LVLMs more effectively than human-feedback baselines.

  4. Once-For-All: A Train-Once and Select-Anytime Framework for Multimodal Instruction Tuning

    cs.CV 2026-05 unverdicted novelty 7.0

    A selector trained once on LLaVA-665K in CLIP space selects 15% of instructions to reach 98.3% of full-data performance and generalizes to an unseen dataset and different VLMs.

  5. FTibSuite: A Comprehensive Resource Suite for Tibetan Vision-Language Modeling

    cs.CV 2026-05 unverdicted novelty 7.0

    FTibSuite provides human-verified multimodal corpora, Tibetan-adapted benchmarks with quality controls, and a baseline VLM showing gains on tasks like MMBench while preserving Chinese capabilities.

  6. SDGBiasBench: Benchmarking and Mitigating Vision--Language Models' Biases in Sustainable Development Goals

    cs.CV 2026-05 unverdicted novelty 7.0

    SDGBiasBench reveals intrinsic SDG biases in VLMs driven by priors rather than evidence, and CADE mitigates them with up to 25% accuracy gains and 12-point MAE reductions.

  7. Are Tools Always Beneficial? Learning to Invoke Tools Adaptively for Dual-Mode Multimodal LLM Reasoning

    cs.CL 2026-05 conditional novelty 7.0

    AutoTool uses reinforcement learning with dual-mode rewards to train multimodal LLMs to adaptively choose between tool-assisted and text-centric reasoning, yielding accuracy and efficiency gains on V* and POPE benchmarks.

  8. Hallucination as Exploit: Evidence-Carrying Multimodal Agents

    cs.AI 2026-05 unverdicted novelty 7.0

    Evidence-carrying multimodal agents decompose tool calls into predicates verified by constrained DOM/OCR/AX checkers to block hallucination-enabled unsafe actions.

  9. UniVLR: Unifying Text and Vision in Visual Latent Reasoning for Multimodal LLMs

    cs.CV 2026-05 unverdicted novelty 7.0

    UniVLR unifies textual and visual reasoning in multimodal LLMs by compressing reasoning traces and auxiliary images into visual latent tokens for direct inference without interleaved text CoT.

  10. CXR-ContraBench: Benchmarking Negated-Option Attraction in Medical VLMs

    cs.CV 2026-05 conditional novelty 7.0

    Medical VLMs frequently select negated options that contradict visible chest X-ray findings, achieving only ~30% accuracy on direct presence probes, but a post-hoc consistency verifier raises accuracy above 95%.

  11. Erase Persona, Forget Lore: Benchmarking Multimodal Copyright Unlearning in Large Vision Language Models

    cs.CV 2026-05 unverdicted novelty 7.0

    CoVUBench is the first benchmark framework for evaluating multimodal copyright unlearning in LVLMs via synthetic data, systematic variations, and a dual protocol for forgetting efficacy and utility preservation.

  12. Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance

    cs.CV 2026-05 unverdicted novelty 7.0

    Gromov-Wasserstein distance between modalities provides a stronger, inference-only predictor of final VLM performance than conventional encoder metrics, backed by theory linking it to cross-modal learnability and veri...

  13. Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models

    cs.CV 2026-04 conditional novelty 7.0

    Prefill-Time Intervention (PTI) reduces hallucinations in large vision-language models by applying a one-time modality-aware steering correction to the initial KV cache at the prefill stage rather than during autoregr...

  14. Improving Vision-language Models with Perception-centric Process Reward Models

    cs.CV 2026-04 unverdicted novelty 7.0

    Perceval is a perception-centric PRM that detects token-level perceptual errors in VLMs, supporting token-advantage RL training and iterative test-time scaling for improved reasoning.

  15. LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models

    cs.CV 2026-04 unverdicted novelty 7.0

    LearnPruner prunes vision tokens to 5.5% of the original count while retaining about 95% of VLM performance and delivering 3.2 times faster inference by fixing attention sink in encoders and using unbiased middle-laye...

  16. ProjLens: Unveiling the Role of Projectors in Multimodal Model Safety

    cs.CR 2026-04 unverdicted novelty 7.0

    ProjLens shows that backdoor parameters in MLLMs are encoded in low-rank subspaces of the projector and that embeddings shift toward the target direction with magnitude linear in input norm, activating only on poisone...

  17. Gaslight, Gatekeep, V1-V3: Early Visual Cortex Alignment Shields Vision-Language Models from Sycophantic Manipulation

    cs.CV 2026-04 conditional novelty 7.0

    Alignment of vision-language models with human V1-V3 early visual cortex negatively predicts resistance to sycophantic gaslighting attacks.

  18. ID-Selection: Importance-Diversity Based Visual Token Selection for Efficient LVLM Inference

    cs.CV 2026-04 unverdicted novelty 7.0

    ID-Selection combines importance scoring with iterative diversity suppression to prune 97.2% of visual tokens in LVLMs while retaining 91.8% performance and cutting FLOPs by over 97% without retraining.

  19. AIA: Rethinking Architecture Decoupling Strategy In Unified Multimodal Model

    cs.CV 2025-11 unverdicted novelty 7.0

    AIA loss teaches unified multimodal models task-specific cross-modal attention patterns to reduce conflicts between image understanding and generation without architecture decoupling.

  20. V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Models

    cs.CL 2025-09 conditional novelty 7.0

    V-SEAM combines concept-level visual semantic editing with attention head modulation to identify positive and negative contributors across object, attribute, and relationship levels, then uses this to improve VLM perf...

  21. MMSearch-R1: Incentivizing LMMs to Search

    cs.CV 2025-06 unverdicted novelty 7.0

    MMSearch-R1 uses reinforcement learning to train multimodal models for on-demand multi-turn internet search with image and text tools, outperforming same-size RAG baselines and matching larger ones while cutting searc...

  22. VGR: Visual Grounded Reasoning

    cs.CV 2025-06 unverdicted novelty 7.0

    VGR introduces a visual-grounded reasoning MLLM that detects and replays image regions during inference, achieving gains on visual benchmarks with 30% fewer image tokens than the LLaVA-NeXT-7B baseline.

  23. DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies

    cs.CV 2025-03 unverdicted novelty 7.0

    DualToken disentangles semantics and appearance via separate codebooks in one tokenizer, reporting 0.25 rFID, 82% ImageNet zero-shot accuracy, and gains over VILA-U on understanding and generation benchmarks.

  24. Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation

    cs.CV 2024-10 unverdicted novelty 7.0

    Janus decouples visual encoding into task-specific pathways inside a single autoregressive transformer to unify multimodal understanding and generation while outperforming earlier unified models.

  25. Detecting and Evaluating Medical Hallucinations in Large Vision Language Models

    cs.CV 2024-06 unverdicted novelty 7.0

    Presents Med-HallMark benchmark, MediHall Score metric, and MediHallDetector model for hallucination detection and evaluation in medical LVLMs.

  26. HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models

    cs.CV 2023-10 unverdicted novelty 7.0

    HallusionBench shows GPT-4V reaches only 31.42% accuracy on paired questions testing language hallucination and visual illusion in LVLMs, with other models below 16%.

  27. LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

    cs.CV 2023-03 conditional novelty 7.0

    LLaMA-Adapter turns frozen LLaMA 7B into a capable instruction follower using only 1.2M new parameters and zero-init attention, matching Alpaca while extending to image-conditioned reasoning on ScienceQA and COCO.

  28. SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference

    cs.LG 2026-06 unverdicted novelty 6.0

    SharQ combines input-adaptive N:M sparsity and FP4 quantization via sparse backbone plus dense residual, recovering 43-63% of the NVFP4-to-FP16 accuracy gap on Llama and Qwen models without calibration or retraining.

  29. The Inattentional Gap: Task-Conditioned Language and Vision Models Omit the Safety-Critical Signals They Can Otherwise Report

    cs.CL 2026-06 unverdicted novelty 6.0

    Task conditioning suppresses safety-critical signal reporting in language and vision models that unconstrained versions report at higher rates, creating an inattentional gap that decouples benchmark safety from real-w...

  30. HPP: Hierarchical Programmatic Probing for Long Video Understanding by Decoupling Perception and Reasoning

    cs.CV 2026-06 unverdicted novelty 6.0

    HPP decouples perception from reasoning in long-video VLMs by having an LLM run iterative programmatic probes on hierarchically segmented video, reporting gains on LongVideoBench, EgoSchema, VideoMME, and MLVU.

  31. Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation

    cs.CV 2026-06 unverdicted novelty 6.0

    Qwen-RobotWorld is a language-conditioned video world model using Double-Stream MMDiT, an 8.6M-frame embodied corpus, and progressive curriculum training that ranks first on EWMBench and DreamGen Bench.

  32. HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizers

    cs.CV 2026-06 unverdicted novelty 6.0

    HYDRA-X presents the first unified multimodal model using a single ViT for holistic image-video tokenization, with ablations on attention and compression plus a latent-level editing improvement.

  33. IDEAL: In-DEpth ALignment Makes A Discrete Representation AutoEncoder

    cs.CV 2026-06 unverdicted novelty 6.0

    IDEAL improves discrete representation autoencoders by jointly aligning quantized tokens with shallow and deep VFM features, reporting 0.61 rFID on ImageNet and 1.89 gFID for autoregressive image generation.

  34. The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 6.0

    FineSightBench reveals VLMs perceive patterns down to 12px but show persistent failures in fine-scale reasoning such as numeracy and sequencing.

  35. Investigating Adversarial Robustness of Multi-modal Large Language Models

    cs.CV 2026-06 unverdicted novelty 6.0

    Robust vision encoders from multimodal adversarial pretraining transfer to MLLMs and deliver large gains in adversarial captioning and VQA performance, while test-time stochastic transformations provide an effective b...

  36. When Attention Collapses: Stage-Aware Visual Token Pruning from Structure to Semantics

    cs.CV 2026-06 unverdicted novelty 6.0

    STS is a two-stage pruning framework that decouples structural diversity via repulsion sampling from semantic filtering via cross-attention to reduce redundancy in visual tokens for VLMs.

  37. Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation

    cs.CV 2026-06 unverdicted novelty 6.0

    TLVS mitigates hallucinations in LVLMs via token-level extraction and visual-sensitivity-adaptive steering applied only at critical decoding steps.

  38. MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention

    cs.CV 2026-06 unverdicted novelty 6.0

    MOSS-Video-Preview introduces a cross-attention architecture and synthesized real-time QA data to enable continuous perception, answer revision, and faster inference in video-language models compared to decoder-only designs.

  39. Attend to Evidence: Evidence-Anchored Spatial Attention Supervision for Multimodal RLVR

    cs.CV 2026-05 unverdicted novelty 6.0

    EASE augments multimodal RLVR with evidence-anchored spatial attention supervision using privileged annotations, improving average benchmark scores by 2.5-3.1 points over DAPO on Qwen VL models.

  40. Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Models

    cs.CV 2026-05 conditional novelty 6.0

    SPpruner reduces visual tokens in VLMs via focus identification followed by context-aware scanning, retaining 22.2% tokens for 2.53x speedup on Qwen2.5-VL with negligible accuracy loss.

  41. Are Tools Always Beneficial? Learning to Invoke Tools Adaptively for Dual-Mode Multimodal LLM Reasoning

    cs.CL 2026-05 unverdicted novelty 6.0

    AutoTool uses dual-mode RL to let MLLMs adaptively choose tool use or text-only reasoning, reporting 21.8% accuracy gain on V* and 44.9% efficiency gain on POPE versus baselines.

  42. Hallucination as Exploit: Evidence-Carrying Multimodal Agents

    cs.AI 2026-05 unverdicted novelty 6.0

    Evidence-carrying multimodal agents decompose tool calls into predicates, obtain certificates from DOM/OCR/AX verifiers, and use a deterministic gate to authorize actions only when certificates support them, achieving...

  43. Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models

    cs.CV 2026-05 unverdicted novelty 6.0

    VIF is a new inference-time module that maintains visual grounding in MLLMs by directly bridging pure visual representations to the output space throughout generation.

  44. Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models

    cs.CV 2026-05 unverdicted novelty 6.0

    Vision Inference Former adds a direct visual-to-output bridge that continuously injects visual semantics during MLLM decoding to sustain consistency and reduce modality imbalance.

  45. Attention Hijacking: Response Manipulation Across Queries in Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 6.0

    Attention Hijacking is a new attack that improves cross-query transferability in VLMs by explicitly steering internal attention to a persistent image-dominant pattern.

  46. From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Grounding

    cs.CV 2026-05 unverdicted novelty 6.0

    A group-revision paradigm for GRPO-based RL fine-tuning of VLMs converts failure responses into improvement signals that refine rewards and advantages, yielding gains on referring segmentation, REC, and counting benchmarks.

  47. LRCP: Low-Rank Compressibility Guided Visual Token Pruning for Efficient LVLMs

    cs.CV 2026-05 unverdicted novelty 6.0

    LRCP prunes visual tokens in LVLMs by scoring projection residuals onto a PCA-estimated low-rank subspace, achieving 88.9% image token reduction with 94.7% performance retention and 87.5% video reduction with 97.8% ac...

  48. GRIP-VLM: Group-Relative Importance Pruning for Efficient Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 6.0

    GRIP-VLM applies group-relative policy optimization via reinforcement learning to prune visual tokens in VLMs, yielding up to 15% inference speedup at matched accuracy over prior methods.

  49. Vocabulary Hijacking in LVLMs: Unveiling Critical Attention Heads by Excluding Inert Tokens to Mitigate Hallucination

    cs.MM 2026-05 unverdicted novelty 6.0

    LVLMs show vocabulary hijacking by inert tokens that decode to hijacking anchors; HABI locates them, NHAR finds resilient heads, and HAVAE boosts those heads to cut hallucinations.

  50. CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering

    cs.CV 2026-05 unverdicted novelty 6.0

    CAST reduces object hallucination in LVLMs by 6.03% on average across five models and five benchmarks by identifying caption-sensitive attention heads and applying optimized steering directions to their outputs, with ...

  51. Online Self-Calibration Against Hallucination in Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 6.0

    OSCAR exploits the generative-discriminative gap in LVLMs to build online preference data with MCTS and dual-granularity rewards for DPO-based calibration, claiming SOTA hallucination reduction and improved multimodal...

  52. X2SAM: Any Segmentation in Images and Videos

    cs.CV 2026-04 unverdicted novelty 6.0

    X2SAM unifies any-segmentation across images and videos in one MLLM by adding a Mask Memory module for temporal consistency and joint training on mixed datasets.

  53. Mitigating Multimodal Hallucination via Phase-wise Self-reward

    cs.CV 2026-04 unverdicted novelty 6.0

    PSRD mitigates visual hallucinations in LVLMs via phase-wise self-reward decoding, cutting rates by 50% on LLaVA-1.5-7B and outperforming prior methods on five benchmarks.

  54. Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models

    cs.CV 2026-04 unverdicted novelty 6.0

    A 0.5B student VLM distills from a 3B teacher using visual-switch distillation and DBiLD loss to gain 3.6 points on average across 10 multimodal benchmarks without architecture changes.

  55. Chain-of-Models Pre-Training: Rethinking Training Acceleration of Vision Foundation Models

    cs.CV 2026-04 unverdicted novelty 6.0

    CoM-PT trains vision foundation models in ascending size order using inverse knowledge transfer, allowing larger models to achieve superior performance with significantly reduced overall computational cost compared to...

  56. Decoupled Similarity for Task-Aware Token Pruning in Large Vision-Language Models

    cs.CV 2026-04 unverdicted novelty 6.0

    DeSAP uses decoupled cross-modal similarity plus visual saliency to prune visual tokens in LVLMs, retaining 11.1% tokens for 10x FLOPs reduction and 98.1% performance on LLaVA-1.5-7B.

  57. From Perception to Planning: Evolving Ego-Centric Task-Oriented Spatiotemporal Reasoning via Curriculum Learning

    cs.AI 2026-04 unverdicted novelty 6.0

    EgoTSR applies a three-stage curriculum on a 46-million-sample dataset to build egocentric spatiotemporal reasoning, reaching 92.4% accuracy on long-horizon tasks and reducing chronological biases.

  58. HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Models

    cs.CV 2026-04 unverdicted novelty 6.0

    HAWK is a training-free method that prunes over 80% of visual tokens in MLLMs while retaining 96% accuracy by using head importance weights and text-guided attention to select task-relevant tokens.

  59. HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models

    cs.CV 2026-04 unverdicted novelty 6.0

    HaloProbe introduces a Bayesian method to estimate token-level hallucination probabilities in VLMs by factorizing external and internal signals, enabling more effective mitigation than intervention-based techniques wh...

  60. Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward

    cs.CV 2026-04 unverdicted novelty 6.0

    Saliency-R1 uses a novel saliency map technique and GRPO with human bounding-box overlap as reward to improve VLM reasoning faithfulness and interpretability.

Reference graph

Works this paper leans on

40 extracted references · 40 canonical work pages · cited by 114 Pith papers · 12 internal anchors

  1. [1]

    Harsh Agrawal, Peter Anderson, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. https://doi.org/10.1109/ICCV.2019.00904 nocaps: novel object captioning at scale . In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019 , pa...

    work page doi:10.1109/iccv.2019.00904 2019

  2. [2]

    Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Kar \' e n Simonyan

    Jean - Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Bin...

    work page 2022

  3. [3]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015 a . VQA: visual question answering. In ICCV , pages 2425--2433. IEEE Computer Society

    work page 2015

  4. [4]

    Lawrence and Parikh, Devi , month = dec, year =

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015 b . https://doi.org/10.1109/ICCV.2015.279 VQA: visual question answering . In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015 , pages 2425--2433. IEEE Computer Society

    work page doi:10.1109/iccv.2015.279 2015

  5. [5]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. CoRR, abs/2308.12966

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  6. [6]

    A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity

    Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. https://doi.org/10.48550/arXiv.2302.04023 A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity . CoRR, abs/2302.04023

    work page internal anchor Pith review doi:10.48550/arxiv.2302.04023 2023

  7. [7]

    Ali Furkan Biten, Llu \' s G \' o mez, and Dimosthenis Karatzas. 2022. https://doi.org/10.1109/WACV51458.2022.00253 Let there be a clock on the beach: Reducing object hallucination in image captioning . In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022, Waikoloa, HI, USA, January 3-8, 2022 , pages 2473--2482. IEEE

    work page doi:10.1109/wacv51458.2022.00253 2022

  8. [8]

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023. Shikra: Unleashing multimodal llm's referential dialogue magic. CoRR, abs/2306.15195

    work page internal anchor Pith review arXiv 2023

  9. [9]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. https://lmsys.org/blog/2023-03-30-vicuna/ Vicuna: An open-source chatbot impressing gpt-4 with 90\

    work page 2023

  10. [10]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023 a . Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500

    work page internal anchor Pith review arXiv 2023

  11. [11]

    Wenliang Dai, Zihan Liu, Ziwei Ji, Dan Su, and Pascale Fung. 2023 b . https://aclanthology.org/2023.eacl-main.156 Plausible may not be faithful: Probing object hallucination in vision-language pre-training . In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-...

    work page 2023

  12. [12]

    Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, and Jianfeng Gao. 2022. https://doi.org/10.1561/0600000105 Vision-language pre-training: Basics, recent advances, and future trends . Found. Trends Comput. Graph. Vis., 14(3-4):163--352

    work page doi:10.1561/0600000105 2022

  13. [13]

    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. 2023. https://doi.org/10.48550/arXiv.2304.15010 Llama-adapter V2: parameter-efficient visual instruction model . CoRR, abs/2304.15010

    work page internal anchor Pith review doi:10.48550/arxiv.2304.15010 2023

  14. [14]

    Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. 2023. Multimodal-gpt: A vision and language model for dialogue with humans. arXiv preprint arXiv:2305.04790

    work page arXiv 2023

  15. [15]

    Yash Goyal, Tejas Khot, Douglas Summers - Stay, Dhruv Batra, and Devi Parikh. 2017. https://doi.org/10.1109/CVPR.2017.670 Making the V in VQA matter: Elevating the role of image understanding in visual question answering . In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , pages 6325--6334....

    work page doi:10.1109/cvpr.2017.670 2017

  16. [16]

    Micah Hodosh, Peter Young, and Julia Hockenmaier. 2015. Framing image description as a ranking task: Data, models and evaluation metrics (extended abstract). In IJCAI , pages 4188--4192. AAAI Press

    work page 2015

  17. [17]

    Yi - Chong Huang, Xia - Chong Feng, Xiao - Cheng Feng, and Bing Qin. 2021. http://arxiv.org/abs/2104.14839 The factual inconsistency problem in abstractive text summarization: A survey . CoRR, abs/2104.14839

    work page arXiv 2021

  18. [18]

    Jackendoff, R.Foundations of Language: Brain, Mean- ing, Grammar, Evolution

    Drew A. Hudson and Christopher D. Manning. 2019. https://doi.org/10.1109/CVPR.2019.00686 GQA: A new dataset for real-world visual reasoning and compositional question answering . In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019 , pages 6700--6709. Computer Vision Foundation / IEEE

    work page doi:10.1109/cvpr.2019.00686 2019

  19. [19]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022. http://arxiv.org/abs/2202.03629 Survey of hallucination in natural language generation . CoRR, abs/2202.03629

    work page arXiv 2022

  20. [20]

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. 2023 a . https://doi.org/10.48550/arXiv.2305.03726 Otter: A multi-modal model with in-context instruction tuning . CoRR, abs/2305.03726

    work page internal anchor Pith review doi:10.48550/arxiv.2305.03726 2023

  21. [21]

    Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven C. H. Hoi. 2022 a . https://doi.org/10.48550/arXiv.2209.09019 LAVIS: A library for language-vision intelligence . CoRR, abs/2209.09019

    work page doi:10.48550/arxiv.2209.09019 2022

  22. [22]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023 b . https://doi.org/10.48550/arXiv.2301.12597 BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models . CoRR, abs/2301.12597

    work page internal anchor Pith review doi:10.48550/arxiv.2301.12597 2023

  23. [23]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022 b . https://proceedings.mlr.press/v162/li22n.html BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation . In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Mac...

    work page 2022

  24. [24]

    Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020. https://doi.org/10.1007/978-3-030-58577-8\_8 Oscar: Object-semantics aligned pre-training for vision-language tasks . In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2...

    work page doi:10.1007/978-3-030-58577-8 2020

  25. [25]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C

    Tsung - Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \' a r, and C. Lawrence Zitnick. 2014. https://doi.org/10.1007/978-3-319-10602-1\_48 Microsoft COCO: common objects in context . In Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V , vo...

    work page doi:10.1007/978-3-319-10602-1 2014

  26. [26]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. https://doi.org/10.48550/arXiv.2304.08485 Visual instruction tuning . CoRR, abs/2304.08485

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2304.08485 2023

  27. [27]

    Bennett, Meredith Ringel Morris, and Edward Cutrell

    Haley MacLeod, Cynthia L. Bennett, Meredith Ringel Morris, and Edward Cutrell. 2017. https://doi.org/10.1145/3025453.3025814 Understanding blind people's experiences with computer-generated captions of social media images . In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, CO, USA, May 06-11, 2017 , pages 5988--5999. ACM

    work page doi:10.1145/3025453.3025814 2017

  28. [28]

    OpenAI. 2023. https://doi.org/10.48550/arXiv.2303.08774 GPT-4 technical report . CoRR, abs/2303.08774

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2303.08774 2023

  29. [29]

    Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. 2011. Im2text: Describing images using 1 million captioned photographs. In NIPS , pages 1143--1151

    work page 2011

  30. [30]

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. https://doi.org/10.18653/v1/d18-1437 Object hallucination in image captioning . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 4035--4045. Association for Computational...

    work page doi:10.18653/v1/d18-1437 2018

  31. [31]

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. https://doi.org/10.1007/978-3-031-20074-8\_9 A-OKVQA: A benchmark for visual question answering using world knowledge . In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VIII , volume 13668 of ...

    work page doi:10.1007/978-3-031-20074-8 2022

  32. [32]

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL (1) , pages 2556--2565. Association for Computational Linguistics

    work page 2018

  33. [33]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie - Anne Lachaux, Timoth \' e e Lacroix, Baptiste Rozi \` e re, Naman Goyal, Eric Hambro, Faisal Azhar, Aur \' e lien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  34. [34]

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. https://proceedings.mlr.press/v162/wang22al.html OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework . In International Conference on Machine Learning, ICML 2022, 17-23 July ...

    work page 2022

  35. [35]

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. 2023. https://doi.org/10.48550/arXiv.2304.14178 mplug-owl: Modularization empowers large language models with multimodality . CoRR, abs/2304.14178

    work page Pith review doi:10.48550/arxiv.2304.14178 2023

  36. [36]

    Peng Zhang, Yash Goyal, Douglas Summers - Stay, Dhruv Batra, and Devi Parikh. 2016. Yin and yang: Balancing and answering binary visual questions. In CVPR , pages 5014--5022. IEEE Computer Society

    work page 2016

  37. [37]

    Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. https://doi.org/10.1109/CVPR46437.2021.00553 Vinvl: Revisiting visual representations in vision-language models . In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021 , pages 5579--5588. Computer ...

    work page doi:10.1109/cvpr46437.2021.00553 2021

  38. [38]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian - Yun Nie, and Ji - Rong Wen. 2023. https://doi.org/10.48550/arXiv.2303.18223 A survey of large langua...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2303.18223 2023

  39. [39]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. https://doi.org/10.48550/arXiv.2304.10592 Minigpt-4: Enhancing vision-language understanding with advanced large language models . CoRR, abs/2304.10592

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2304.10592 2023

  40. [40]

    Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. 2023. https://doi.org/10.48550/arXiv.2304.06718 Segment everything everywhere all at once . CoRR, abs/2304.06718

    work page doi:10.48550/arxiv.2304.06718 2023