pith. sign in

arxiv: 2302.00923 · v5 · submitted 2023-02-02 · 💻 cs.CL · cs.AI· cs.CV

Multimodal Chain-of-Thought Reasoning in Language Models

Zhuosheng Zhang , Aston Zhang , Mu Li , Hai Zhao , George Karypis , Alex Smola This is my paper

Pith reviewed 2026-05-12 18:08 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.CV
keywords multimodal chain-of-thoughtlanguage modelsScienceQArationale generationanswer inferencevision-language reasoninghallucination
0
0 comments X

The pith

Multimodal-CoT separates rationale generation from answer inference to enable state-of-the-art reasoning in small language models using both text and images.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Multimodal-CoT, which extends chain-of-thought prompting to include visual inputs through a two-stage process. In the first stage, the model generates rationales grounded in both language and vision data. The second stage then uses those rationales to infer the final answer. This structure allows smaller models, under one billion parameters, to outperform larger ones on the ScienceQA benchmark. The approach also helps reduce hallucinations and speeds up model convergence.

Core claim

By separating the generation of multimodal rationales from the subsequent answer inference step, language models can leverage richer information from images and text to produce more accurate reasoning chains, resulting in state-of-the-art performance on ScienceQA with models under 1 billion parameters.

What carries the argument

The two-stage framework separating rationale generation (using multimodal inputs) from answer inference.

If this is right

  • Answer inference benefits from higher-quality rationales informed by both modalities.
  • The method mitigates hallucination in generated outputs.
  • Training convergence is accelerated compared to standard approaches.
  • Strong results transfer to the A-OKVQA benchmark as well.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar separation of reasoning steps could improve performance on other vision-language tasks not tested here.
  • Smaller models might close the gap with larger ones across more benchmarks if this two-stage pattern is adopted broadly.
  • Explicit stage separation may serve as a general technique to enhance chain-of-thought reliability in multimodal settings.

Load-bearing premise

That separating rationale generation from answer inference will reliably produce higher-quality multimodal rationales without introducing new error modes or requiring task-specific tuning that offsets the gains.

What would settle it

A controlled experiment showing that a model using the two-stage Multimodal-CoT does not outperform an equivalent single-stage multimodal prompting baseline on ScienceQA accuracy.

read the original abstract

Large language models (LLMs) have shown impressive performance on complex reasoning by leveraging chain-of-thought (CoT) prompting to generate intermediate reasoning chains as the rationale to infer the answer. However, existing CoT studies have primarily focused on the language modality. We propose Multimodal-CoT that incorporates language (text) and vision (images) modalities into a two-stage framework that separates rationale generation and answer inference. In this way, answer inference can leverage better generated rationales that are based on multimodal information. Experimental results on ScienceQA and A-OKVQA benchmark datasets show the effectiveness of our proposed approach. With Multimodal-CoT, our model under 1 billion parameters achieves state-of-the-art performance on the ScienceQA benchmark. Our analysis indicates that Multimodal-CoT offers the advantages of mitigating hallucination and enhancing convergence speed. Code is publicly available at https://github.com/amazon-science/mm-cot.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The paper proposes Multimodal-CoT, a two-stage framework for chain-of-thought reasoning in language models that incorporates both text and image modalities. The first stage generates multimodal rationales, and the second stage performs answer inference using those rationales. Experiments on the ScienceQA and A-OKVQA benchmarks demonstrate performance gains, with a model under 1 billion parameters achieving state-of-the-art results on ScienceQA; additional analysis claims reduced hallucination and faster convergence. The code is released publicly.

Significance. If the empirical results hold, the work shows that separating rationale generation from answer inference can improve multimodal reasoning performance even for sub-billion-parameter models, extending CoT techniques beyond language-only settings. Public code availability aids reproducibility and enables further exploration of hallucination mitigation in vision-language tasks.

major comments (1)
  1. [§4 (Experiments)] §4 (Experiments): The central claim that the two-stage separation produces higher-quality multimodal rationales rests on overall benchmark gains, but without full ablation tables isolating the contribution of rationale generation versus joint multimodal prompting (or single-stage baselines), it is difficult to rule out that gains arise from other factors such as prompt engineering or training details.
minor comments (2)
  1. [Abstract / §1] The abstract and §1 could more precisely quantify the SOTA margin on ScienceQA (e.g., absolute accuracy delta versus prior best) and specify the exact model architecture and parameter count used.
  2. [Figure 2] Figure 2 and associated text would benefit from clearer labeling of the two stages and explicit comparison of rationale quality metrics (e.g., human or automatic evaluation of rationale faithfulness).

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the positive evaluation of our work and the constructive comment on the experimental section. We address the major comment below.

read point-by-point responses

  1. Referee: [§4 (Experiments)] §4 (Experiments): The central claim that the two-stage separation produces higher-quality multimodal rationales rests on overall benchmark gains, but without full ablation tables isolating the contribution of rationale generation versus joint multimodal prompting (or single-stage baselines), it is difficult to rule out that gains arise from other factors such as prompt engineering or training details.

    Authors: We appreciate this point and agree that stronger isolation of the two-stage design would further substantiate the central claim. The current manuscript includes ablation studies in Section 4.3 that compare the full two-stage Multimodal-CoT against single-stage multimodal baselines (direct answer inference without separate rationale generation) as well as language-only CoT variants. These results show consistent gains from the two-stage separation on ScienceQA, including lower hallucination rates. However, to more rigorously rule out confounds from prompt engineering or training details, we will add expanded ablation tables in the revision. These will include controlled comparisons of rationale generation versus joint multimodal prompting under matched training and prompt conditions. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper proposes Multimodal-CoT as an empirical two-stage prompting framework that separates rationale generation from answer inference to incorporate both text and image modalities. All central claims rest on reported experimental results on the held-out ScienceQA and A-OKVQA benchmarks rather than any mathematical derivation, fitted parameter, or self-referential definition. No equations, uniqueness theorems, or ansatzes are invoked that could reduce the method to its own inputs by construction; the separation of stages is presented as a design choice whose value is measured externally. The work is therefore self-contained as an engineering technique with no load-bearing circular steps.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is an empirical methods paper whose central claim rests on experimental validation of a prompting framework rather than on new axioms, free parameters, or invented entities.

pith-pipeline@v0.9.0 · 5468 in / 1031 out tokens · 31061 ms · 2026-05-12T18:08:05.883026+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Position Rebinding Cache Reuse: Replay-Free Visual Revisiting for Interleaved Multimodal Reasoning

    cs.CV 2026-06 unverdicted novelty 7.0

    PRCR enables replay-free visual revisiting in interleaved multimodal reasoning by storing raw visual KV caches with spatial coordinates and rebinding keys to position-compatible coordinates, matching replay performanc...

  2. TriViewBench: Controlled Complexity Scaling for Multi-View Structural Reasoning in MLLMs

    cs.CV 2026-06 unverdicted novelty 7.0

    TriViewBench shows all tested MLLMs follow the same capability order with sharp drops on complex multi-view tasks and near-zero gain from Chain-of-Thought prompting.

  3. SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs

    cs.CV 2026-06 unverdicted novelty 7.0

    SPOT-E uses entropy shaping on answer predictions with low-entropy anchors to optimize visual spotlights at test time via GRPO for better VLM performance on evidence-intensive tasks.

  4. Are Reasoning Vision-Language Models Robust to Semantic Visual Distractions?

    cs.CV 2026-06 unverdicted novelty 7.0

    Reasoning VLMs show lower robustness to semantic visual distractions than to perceptual corruptions, with distractions entering their reasoning chains and causing errors.

  5. TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding

    cs.CV 2026-06 unverdicted novelty 7.0

    TVI-CoT introduces learnable control tokens <THINK>, <LOOK>, <ANSWER> that let multimodal LLMs interleave textual reasoning with dynamic visual feature access, reporting gains of 3.4-6.1% on eight benchmarks over prio...

  6. Imagine Before You Predict: Interleaved Latent Visual Reasoning for Video Event Prediction

    cs.CV 2026-06 unverdicted novelty 7.0

    Future-L1 interleaves latent visual spans with text in MLLM decoding, trained on a custom Future-L1-50K dataset via LA-DAPO RL, and reports SOTA gains on FutureBench (61.0 to 85.4) and TwiFF-Bench (2.44 to 3.04).

  7. ToolFG: Towards Well-Grounded Fine-Grained Image Classification

    cs.CV 2026-06 unverdicted novelty 7.0

    ToolFG is the first tool-integrated MLLM framework for fine-grained image classification, trained via MCTS-guided distillation from proprietary models and refined through model-tool co-evolution.

  8. Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning

    cs.CV 2026-06 unverdicted novelty 7.0

    Attentive-CoT is an attention-guided fine-tuning objective that improves chain-of-thought performance in multimodal LLMs by delaying answer commitment and increasing sustained visual-token access during rationale generation.

  9. MuCRASP: Multimodal Chain-of-thought Reasoning aware Structured Pruning

    cs.AI 2026-05 unverdicted novelty 7.0

    MuCRASP prunes VLMs in a CoT-aware manner, outperforming baselines by preserving reasoning quality at 30-50% compression rates on models like Qwen2.5-VL-7B.

  10. LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning

    cs.CL 2026-05 unverdicted novelty 7.0

    LatentOmni proposes a latent-space cross-modal reasoning framework that uses feature-level supervision and Omni-Sync Position Embedding to align and synchronize audio-visual latents, supported by a new 35K interleaved...

  11. ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both

    cs.CV 2026-05 unverdicted novelty 7.0

    ATLAS uses a single functional token to unify agentic and latent visual reasoning without image generation or external execution.

  12. V-ABS: Action-Observer Driven Beam Search for Dynamic Visual Reasoning

    cs.CV 2026-05 unverdicted novelty 7.0

    V-ABS is an action-observer beam search method with entropy-based adaptive weighting and an 80k-sample SFT dataset that delivers 19.7% average gains on visual reasoning tasks for MLLMs.

  13. HyLaR: Hybrid Latent Reasoning with Decoupled Policy Optimization

    cs.CV 2026-04 unverdicted novelty 7.0

    HyLaR with DePO enables effective RL in hybrid discrete-continuous spaces for multimodal models, outperforming prior MLLMs on perception and understanding benchmarks.

  14. GeoBrowse: A Geolocation Benchmark for Agentic Tool Use with Expert-Annotated Reasoning Traces

    cs.CL 2026-04 unverdicted novelty 7.0

    GeoBrowse is a two-level geolocation benchmark combining visual cue composition with knowledge-intensive multi-hop queries, paired with the GATE agent workflow that outperforms no-tool, search-only, and image-only baselines.

  15. Reasoning Within the Mind: Dynamic Multimodal Interleaving in Latent Space

    cs.CV 2025-12 unverdicted novelty 7.0

    DMLR performs dynamic visual-textual interleaving in latent space using confidence-guided latent policy gradient optimization and a dynamic visual injection strategy, yielding improved multimodal reasoning on benchmarks.

  16. Latent Visual Reasoning

    cs.CV 2025-09 unverdicted novelty 7.0

    Latent Visual Reasoning enables autoregressive generation of latent visual states that reconstruct critical image tokens, yielding gains on perception-heavy VQA benchmarks such as 71.67% on MMVP.

  17. From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systems

    cs.MA 2025-06 accept novelty 7.0

    A survey that defines Compound AI Systems, proposes a multi-dimensional taxonomy based on component roles and orchestration strategies, reviews four foundational paradigms, and identifies key challenges for future research.

  18. Rethinking Information Synthesis in Multimodal Question Answering A Multi-Agent Perspective

    cs.CL 2025-05 unverdicted novelty 7.0

    MAMMQA is a multi-agent framework that decomposes multimodal queries, retrieves modality-specific answers, performs cross-modal synthesis with VLMs, and integrates results via an LLM to outperform single-model baselin...

  19. GRIT: Teaching MLLMs to Think with Images

    cs.CV 2025-05 unverdicted novelty 7.0

    GRIT introduces a grounded reasoning paradigm for MLLMs where reasoning chains interleave text and bounding boxes, trained via GRPO-GR reinforcement learning on as few as 20 examples without annotations.

  20. Video-R1: Reinforcing Video Reasoning in MLLMs

    cs.CV 2025-03 conditional novelty 7.0

    Video-R1 uses temporal-aware RL and mixed datasets to boost video reasoning in MLLMs, with a 7B model reaching 37.1% on VSI-Bench and surpassing GPT-4o.

  21. Visual Instruction Tuning

    cs.CV 2023-04 unverdicted novelty 7.0

    LLaVA is trained on GPT-4 generated visual instruction data to achieve 85.1% relative performance to GPT-4 on synthetic multimodal tasks and 92.53% accuracy on Science QA.

  22. LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

    cs.CV 2023-03 conditional novelty 7.0

    LLaMA-Adapter turns frozen LLaMA 7B into a capable instruction follower using only 1.2M new parameters and zero-init attention, matching Alpaca while extending to image-conditioned reasoning on ScienceQA and COCO.

  23. Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models

    cs.CV 2023-03 accept novelty 7.0

    Visual ChatGPT integrates visual foundation models with ChatGPT via prompts to enable multi-step image understanding, generation, and editing in conversational interactions.

  24. Multimodal Continuous Reasoning via Asymmetric Mutual Variational Learning

    cs.CV 2026-07 unverdicted novelty 6.0

    AMVL applies bidirectional KL calibration to align answer-agnostic prior with answer-conditioned posterior in variational multimodal reasoning, reducing leakage and yielding +10.83 average gain on BLINK benchmark.

  25. ReShift: Aha-Moment-Driven Reasoning-Level Backdoor Attacks on Vision-Language Models

    cs.CR 2026-07 unverdicted novelty 6.0

    ReShift is a reasoning-level backdoor framework for VLMs that uses poisoned data construction and joint optimization to shift CoT trajectories on trigger while preserving surface coherence.

  26. CoLT: Teaching Multi-Modal Models to Think with Chain of Latent Thoughts

    cs.CV 2026-06 unverdicted novelty 6.0

    CoLT replaces text-based chain-of-thought in MLLMs with 3-step latent thought chains supervised by a removable external decoder in forward and backward modes, yielding 10.1x faster inference on eight benchmarks.

  27. Latent Noise Mask for Reducing Visual Redundancy in Multimodal Large Language Models

    cs.CV 2026-06 unverdicted novelty 6.0

    Lens purifies visual evidence in MLLMs via question-conditioned latent noise masking with a LET token, yielding 2.4-6.4 point gains on VQA and grounding tasks.

  28. CVSBench: A Comprehensive Benchmark for Cross-view Spatial Reasoning and Dreaming

    cs.CV 2026-06 unverdicted novelty 6.0

    CVSBench benchmark shows VLMs struggle with cross-view spatial consistency but improve substantially when given 3D scene imagination inputs.

  29. CARE: Competence-Aware Reward Shaping for Adaptive Reasoning Length in Video-MLLMs

    cs.CV 2026-06 unverdicted novelty 6.0

    CARE uses exponential moving average competence estimates to progressively shift RL rewards from exploration-oriented long reasoning to efficiency-oriented concise reasoning in video-MLLMs, with batch normalization an...

  30. Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning

    cs.CV 2026-06 unverdicted novelty 6.0

    Visual-OPSD distills reasoning from a privileged visual-thought teacher to a text-only student using on-policy JSD, delivering +3.40pp accuracy gain and 14.3x speedup over the generative teacher on nine benchmarks.

  31. RoboPIN: Grounded Embodied Reasoning via Pinned Chain-of-Thought

    cs.AI 2026-06 unverdicted novelty 6.0

    Introduces PinCoT paradigm with visual reasoning anchors, builds PIN-170K dataset via automated pipeline, and trains 4B RoboPIN model via three-stage post-training to outperform 7B baselines by 12% on embodied reasoni...

  32. NeRD: Neuro-Symbolic Rule Distillation for Efficient Ontology-Grounded Chain-of-Thought in Medical Image Diagnosis

    cs.CV 2026-06 unverdicted novelty 6.0

    NeRD framework generates efficient ontology-grounded reasoning chains for medical image diagnosis via neuro-symbolic rule distillation, shown on skin datasets with expert validation.

  33. MUSE: A Unified Agentic Harness for MLLMs

    cs.CV 2026-06 unverdicted novelty 6.0

    MUSE is a unified agentic harness that improves off-the-shelf MLLMs on visual spatial planning, perception, multimodal reasoning, and fine-grained discrimination benchmarks through structured execution modules and ver...

  34. Unveiling the Visual Counting Bottleneck in Vision-Language Models

    cs.MM 2026-05 unverdicted novelty 6.0

    VLMs fail at visual counting extrapolation because they cannot project visual magnitudes onto symbolic tokens, despite intact perceptual representations, supporting a fractured magnitude hypothesis.

  35. ROVER: Routing Object-Centric Visual Evidence for Grounded Multi-Image Reasoning

    cs.CV 2026-05 unverdicted novelty 6.0

    ROVER introduces a learnable routing plugin for object-centric visual evidence in MLLMs via token triplets and differential attention, reporting gains on MM-GCoT and VideoEspresso when integrated into Qwen2.5-VL-7B.

  36. Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization

    cs.AI 2026-05 unverdicted novelty 6.0

    RC-DPO adds a CoT-conditioned preference term to DPO and pairs it with MCTS-based positive CoT generation plus attention-guided pruning for negatives, yielding lower hallucination rates on multimodal benchmarks.

  37. ProSR: Process-Shaped Spatial Reasoning for Reliable Chain-of-Thought in VLMs

    cs.CV 2026-05 unverdicted novelty 6.0

    ProSR adds a Counterfactual Invariance Penalty and a Tail Drift Penalty to shape VLM reasoning trajectories for better visual dependence and stability on spatial tasks.

  38. Can MLLMs Reason About Visual Persuasion? Evaluating the Efficacy and Faithfulness of Reasoning

    cs.CV 2026-05 conditional novelty 6.0

    Diverse teacher-generated rationales improve MLLM visual persuasiveness prediction via supervised fine-tuning, while a new three-dimensional faithfulness framework shows that prediction accuracy alone does not ensure ...

  39. Retrieve, Integrate, and Synthesize: Spatial-Semantic Grounded Latent Visual Reasoning

    cs.CL 2026-05 unverdicted novelty 6.0

    RIS improves MLLM latent visual reasoning by retrieving spatial-semantic evidence, integrating it via attention bottlenecks, and synthesizing it with language transition tokens, yielding gains on V*, HRBench, MMVP, an...

  40. State Beyond Appearance: Diagnosing and Improving State Consistency in Dial-Based Measurement Reading

    cs.CV 2026-04 unverdicted novelty 6.0

    MLLMs ignore dial state geometry and cluster by appearance, causing inconsistency under variations; TriSCA's state-distance alignment, metadata supervision, and objective alignment improve robustness on clock and gaug...

  41. Meta-CoT: Enhancing Granularity and Generalization in Image Editing

    cs.CV 2026-04 unverdicted novelty 6.0

    Meta-CoT uses two-level decomposition of editing operations into meta-tasks and a CoT consistency reward to improve granularity and generalization, reporting 15.8% gains across 21 tasks.

  42. Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Models

    cs.CV 2026-04 unverdicted novelty 6.0

    Foveated Reasoner integrates foveation as stateful actions inside the autoregressive decoding loop of vision-language models, trained via cold-start supervision then reinforcement learning to achieve higher accuracy a...

  43. V-tableR1: Process-Supervised Multimodal Table Reasoning with Critic-Guided Policy Optimization

    cs.AI 2026-04 unverdicted novelty 6.0

    V-tableR1 uses a critic VLM for dense step-level feedback and a new PGPO algorithm to shift multimodal table reasoning from pattern matching to verifiable logical steps, achieving SOTA accuracy with a 4B open-source model.

  44. HyLaR: Hybrid Latent Reasoning with Decoupled Policy Optimization

    cs.CV 2026-04 conditional novelty 6.0

    HyLaR interleaves discrete text generation with continuous visual latent representations and optimizes them via a decoupled RL algorithm using vMF distributions, improving fine-grained visual reasoning.

  45. Think in Latent Thoughts: A New Paradigm for Gloss-Free Sign Language Translation

    cs.CV 2026-04 unverdicted novelty 6.0

    A new SLT framework uses latent thoughts as a middle reasoning layer and plan-then-ground decoding to improve coherence and faithfulness in gloss-free sign language translation.

  46. From Perception to Planning: Evolving Ego-Centric Task-Oriented Spatiotemporal Reasoning via Curriculum Learning

    cs.AI 2026-04 unverdicted novelty 6.0

    EgoTSR applies a three-stage curriculum on a 46-million-sample dataset to build egocentric spatiotemporal reasoning, reaching 92.4% accuracy on long-horizon tasks and reducing chronological biases.

  47. AITP: Traffic Accident Responsibility Allocation via Multimodal Large Language Models

    cs.CL 2026-04 unverdicted novelty 6.0

    AITP is a new multimodal large language model that uses multimodal chain-of-thought and retrieval-augmented generation of legal knowledge to achieve state-of-the-art results on traffic accident responsibility allocati...

  48. MP-ISMoE: Mixed-Precision Interactive Side Mixture-of-Experts for Efficient Transfer Learning

    cs.LG 2026-04 unverdicted novelty 6.0

    MP-ISMoE uses Gaussian noise perturbed iterative quantization and interactive side mixture-of-experts to deliver higher accuracy than prior memory-efficient transfer learning methods while keeping similar parameter an...

  49. Walk the Talk: Bridging the Reasoning-Action Gap for Thinking with Images via Multimodal Agentic Policy Optimization

    cs.CV 2026-04 unverdicted novelty 6.0

    MAPO improves multimodal chain-of-thought reasoning by requiring explicit textual descriptions of visual tool results and using a novel advantage estimator that combines semantic alignment with task rewards.

  50. Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models

    cs.AI 2026-04 unverdicted novelty 6.0

    Position and step penalty plus visual reasoning guidance fix premature answering and weak visual grounding in diffusion MLLMs, delivering up to 7.5% accuracy gains and over 3x speedup.

  51. C2F-Thinker: Coarse-to-Fine Reasoning with Hint-Guided Reinforcement Learning for Multimodal Sentiment Analysis

    cs.CL 2026-03 unverdicted novelty 6.0

    C2F-Thinker combines structured coarse-to-fine chain-of-thought reasoning with hint-guided GRPO reinforcement learning to achieve competitive fine-grained sentiment regression and superior cross-domain generalization ...

  52. Thinking with Drafting: Optical Decompression via Logical Reconstruction

    cs.CL 2026-02 unverdicted novelty 6.0

    Thinking with Drafting reconceptualizes visual reasoning as optical decompression by forcing models to draft mental models into executable DSL code for deterministic self-verification on the VisAlg benchmark.

  53. Multimodal Reinforcement Learning with Adaptive Verifier for AI Agents

    cs.AI 2025-12 unverdicted novelty 6.0

    Argos is an agentic verifier that adaptively picks scoring functions to evaluate accuracy, localization, and reasoning quality, enabling stronger multimodal RL training for AI agents.

  54. FATHOMS-RAG: A Framework for the Assessment of Thinking and Observation in Multimodal Systems that use Retrieval Augmented Generation

    cs.AI 2025-10 unverdicted novelty 6.0

    Introduces a 93-question multimodal RAG benchmark with phrase-level recall and embedding-based hallucination metrics, finding closed-source pipelines outperform open-source ones especially on cross-modal and cross-doc...

  55. AIM-CoT: Active Information-driven Multimodal Chain-of-Thought for Vision-Language Reasoning

    cs.CV 2025-09 unverdicted novelty 6.0

    AIM-CoT enhances interleaved multimodal chain-of-thought reasoning by adding context-enhanced attention generation, active visual probing via information foraging, and dynamic attention-shift triggering.

  56. Mitigating Visual Context Degradation in Large Multimodal Models: A Training-Free Decoupled Agentic Framework

    cs.CV 2025-09 unverdicted novelty 6.0

    DRP decouples reasoning from perception in LMMs by using an LLM reasoner to query an LMM observer for visual details as needed, reducing visual grounding loss.

  57. LaV-CoT: Language-Aware Visual CoT with Multi-Aspect Reward Optimization for Real-World Multilingual VQA

    cs.CV 2025-09 unverdicted novelty 6.0

    LaV-CoT introduces a multi-stage visual CoT pipeline and GRPO training with language-consistency rewards, delivering up to 9.5% accuracy gains on multilingual VQA benchmarks over similar-sized open models.

  58. TaleDiffusion: Multi-Character Story Generation with Dialogue Rendering

    cs.CV 2025-09 unverdicted novelty 6.0

    TaleDiffusion introduces an iterative framework using LLM-generated per-frame descriptions, bounded attention-based per-box masks, identity-consistent self-attention, region-aware cross-attention, and CLIPSeg-based di...

  59. The Landscape of Agentic Reinforcement Learning for LLMs: A Survey

    cs.AI 2025-09 accept novelty 6.0

    Survey that defines agentic RL for LLMs via POMDPs, introduces a taxonomy of planning/tool-use/memory/reasoning capabilities and domains, and compiles open environments from over 500 papers.

  60. PosterForest: Hierarchical Multi-Agent Collaboration for Scientific Poster Generation

    cs.AI 2025-08 unverdicted novelty 6.0

    PosterForest uses a Poster Tree intermediate representation and hierarchical multi-agent reasoning to generate coherent scientific posters without training, outperforming prior methods in evaluations.

Reference graph

Works this paper leans on

49 extracted references · 49 canonical work pages · cited by 92 Pith papers · 10 internal anchors

  1. [1]

    Bottom-up and top-down attention for image captioning and visual question answering

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 6077–6086. IEEE Computer Society,

    work page 2018

  2. [2]

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al

    doi: 10.1109/CVPR.2018.00636. Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models.arXiv preprint arXiv:2308.01390,

    work page doi:10.1109/cvpr.2018.00636 2018

  3. [3]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    work page 2020

  4. [4]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I, pp. 213–229,

    work page 2020

  5. [5]

    Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-supervised models are strong semi-supervised learners. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.),Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing...

    work page 2020

  6. [6]

    Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Dis- entangling computation from reasoning for numerical reasoning tasks.ArXiv preprint, abs/2211.12588,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    See, think, confirm: Interactive prompting between vision and language models for knowledge-based visual reasoning,

    Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Hao Zhang, and Chuang Gan. See, think, confirm: Interactive prompting between vision and language models for knowledge-based visual reasoning.ArXiv preprint, abs/2301.05226,

    work page arXiv

  8. [8]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, 13 Published in Transactions on Machine Learning Research (05/2024) Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabh...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  9. [9]

    Scaling Instruction-Finetuned Language Models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models.ArXiv preprint, abs/2210.11416,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In9th International Conference on Learning Representations, ICLR 2021, V...

    work page 2021

  11. [11]

    OpenReview.net, 2021a. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In9th International Conference on Learning Repres...

    work page 2021

  12. [12]

    arXiv preprint arXiv:2210.00720 , year=

    OpenReview.net, 2021b. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting for multi-step reasoning.ArXiv preprint, abs/2210.00720,

    work page arXiv

  13. [13]

    Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven C. H. Hoi, Xiaogang Wang, and Hongsheng Li. Dynamic fusion with intra- and inter-modality attention flow for visual question answering. InIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 6639–6648. Computer Vision Foundation / IEEE,

    work page 2019

  14. [14]

    Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei

    doi: 10.1109/CVPR.2019.00680. Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. Language models are general-purpose interfaces.ArXiv preprint, abs/2206.06336,

    work page doi:10.1109/cvpr.2019.00680 2019

  15. [15]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770–778. IEEE Computer Society,

    work page 2016

  16. [16]

    Deep residual learning for image recognition

    doi: 10.1109/CVPR.2016.90. Namgyu Ho, Laura Schmid, and Se-Young Yun. Large language models are reasoning teachers.ArXiv preprint, abs/2212.10071,

    work page doi:10.1109/cvpr.2016.90 2016

  17. [17]

    Towards Reasoning in Large Language Models: A Survey

    Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey.ArXiv preprint, abs/2212.10403,

    work page internal anchor Pith review arXiv

  18. [18]

    UNIFIEDQA: Crossing format boundaries with a single QA system

    Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1896–1907, Online,

    work page 2020

  19. [19]

    UNIFIEDQA : Crossing format boundaries with a single QA system

    Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.171. 14 Published in Transactions on Machine Learning Research (05/2024) Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks.ArXiv preprint, abs/2210.02406,

    work page doi:10.18653/v1/2020.findings-emnlp.171 2020

  20. [20]

    Bilinear attention networks

    Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. Bilinear attention networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.),Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Can...

    work page 2018

  21. [21]

    Vilt: Vision-and-language transformer without convolution or region supervision

    Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In Marina Meila and Tong Zhang (eds.),Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 ofProceedings of Machine Learning Research, pp. 5583–5594. PMLR,

    work page 2021

  22. [22]

    Large Language Models are Zero-Shot Reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners.ArXiv preprint, abs/2205.11916,

    work page internal anchor Pith review arXiv

  23. [23]

    On vision features in multimodal machine translation

    Bei Li, Chuanhao Lv, Zefan Zhou, Tao Zhou, Tong Xiao, Anxiang Ma, and JingBo Zhu. On vision features in multimodal machine translation. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6327–6337, Dublin, Ireland, 2022a. Association for Computational Linguistics. doi: 10.18653/v1/2022.ac...

    work page doi:10.18653/v1/2022.acl-long.438 2022

  24. [24]

    Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen

    Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. On the advance of making language models better reasoners.ArXiv preprint, abs/2206.02336, 2022c. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.ArXiv preprint, abs/2304.08485,

    work page arXiv

  25. [25]

    arXiv preprint arXiv:2209.14610 , year=

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022a. Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Ra...

    work page arXiv 2023

  26. [26]

    arXiv preprint arXiv:2212.08410 , year=

    Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason.ArXiv preprint, abs/2212.08410,

    work page arXiv

  27. [27]

    Gpt-4v(ision) system card,

    15 Published in Transactions on Machine Learning Research (05/2024) OpenAI. Gpt-4v(ision) system card,

    work page 2024

  28. [28]

    URL https://openreview.net/forum?id=lLmqxkfSIw. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.),Proceedings of...

    work page 2021

  29. [29]

    Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, ...

    work page internal anchor Pith review Pith/arXiv arXiv

  30. [30]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530,

    work page internal anchor Pith review Pith/arXiv arXiv

  31. [31]

    Learning to retrieve prompts for in-context learning

    Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2655–2671, Seattle, United States,

    work page 2022

  32. [32]

    Learning To Retrieve Prompts for In-Context Learning , url =

    Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.191. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. InComputer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Par...

    work page doi:10.18653/v1/2022.naacl-main.191 2022

  33. [33]

    Alpaca: A strong, replicable instruction-following model.Stanford Center for Research on Foundation Models

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-following model.Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html,

    work page 2023

  34. [34]

    LaMDA: Language Models for Dialog Applications

    Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  35. [35]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.),Advances in Neural Information Processing Systems 30: Annual Conference on...

    work page 2017

  36. [36]

    Self-Consistency Improves Chain of Thought Reasoning in Language Models

    Boshi Wang, Xiang Deng, and Huan Sun. Iteratively prompt pre-trained language models for chain of thought. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2714–2730, Abu Dhabi, United Arab Emirates, 2022a. Association for Computational Linguistics. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  37. [37]

    doi: 10.18653/v1/2021.acl-long.480

    Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.480. Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Retrieval-augmented multimodal language modeling.Proceedings of the 40th International Conference on Machine Learning, PMLR, pp. 39755–39769,

    work page doi:10.18653/v1/2021.acl-long.480 2021

  38. [38]

    Deep modular co-attention networks for visual question answering

    Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. InIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 6281–6290. Computer Vision Foundation / IEEE,

    work page 2019

  39. [39]

    doi: 10.1109/CVPR.2019.00644. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding a...

    work page doi:10.1109/cvpr.2019.00644 2019

  40. [40]

    LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

    Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention.ArXiv preprint, abs/2303.16199, 2023a. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren’...

    work page Pith review arXiv 2024

  41. [41]

    Universal multimodal representation for language understanding.IEEE Transactions on Pattern Analysis and Machine Intelligence, pp

    Zhuosheng Zhang, Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita, Zuchao Li, and Hai Zhao. Universal multimodal representation for language understanding.IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–18, 2023c. doi: 10.1109/TPAMI.2023.3234170. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prom...

    work page doi:10.1109/tpami.2023.3234170 2023

  42. [42]

    Least-to-Most Prompting Enables Complex Reasoning in Large Language Models

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. ArXiv preprint, abs/2205.10625,

    work page internal anchor Pith review Pith/arXiv arXiv

  43. [43]

    Here, we present additional examples to illustrate this phenomenon, as depicted in Figure

    18 Published in Transactions on Machine Learning Research (05/2024) A Extended Analysis for the Challenge of Multimodal-CoT A.1 Additional Examples of Misleading through Hallucinated Rationales Based on our case studies (Section 3.2), we have observed a tendency for the baseline model to generate hallucinated rationales. Here, we present additional exampl...

    work page 2024

  44. [44]

    (C) The magnitude of the magnetic force is the same in both pairs. (C) neither; their concentrations are the same Figure 7: Examples of the two-stage framework without vision features (baseline) and with vision features (ours) for generating rationales and predicting answers. The upper part presents the problem details, and the lower part shows the output...

    work page 2024

  45. [45]

    •ScienceQA is a large-scale multimodal science question dataset with annotated lectures and explanations

    benchmark datasets. •ScienceQA is a large-scale multimodal science question dataset with annotated lectures and explanations. It contains 21k multimodal multiple choice questions with rich domain diversity across 3 subjects, 26 topics, 127 categories, and 379 skills. The dataset is split into training, validation, and test splits with 12k, 4k, and 4k ques...

    work page 2024

  46. [46]

    The vision features are obtained by the frozen ViT-large encoder (Dosovitskiy et al., 2021b)

    (Section 6.1). The vision features are obtained by the frozen ViT-large encoder (Dosovitskiy et al., 2021b). Since using image captions can slightly improve model performance, as shown in Section 3.3, we append the image captions to the context following Lu et al. (2022a). The captions are generated by InstructBLIP (Dai et al., 2023). We fine-tune the mod...

    work page 2023

  47. [47]

    Our experiments are run on 8 NVIDIA Tesla V100 32G GPUs. C Further Analysis C.1 Examples of Rationale Generation with Large Models A recent flame is to leverage large language models or large vision-language models to generate reasoning chains for multimodal question answering problems (Zhang et al., 2023a; Lu et al., 2023; Liu et al., 2023; Alayrac et al...

    work page 2023

  48. [48]

    Then, we use the generated pseudo-rationales as the target rationales for training instead of relying on the human annotation of reasoning chains

    (Figure 10a) and ChatGPT (Figure 10b) for zero-shot inference, respectively. Then, we use the generated pseudo-rationales as the target rationales for training instead of relying on the human annotation of reasoning chains. OutputThegreenarearepresentsthestateofNewHampshire,whichislocatedinthenortheasternregionoftheUnitedStates.Thereareseveralotherstatesv...

    work page 2024

  49. [49]

    80% 14%6% CommonsenseLogicalOthers Figure 11: Categorization analysis

    We examined 50 samples that yielded incorrect answers and categorized them accordingly. 80% 14%6% CommonsenseLogicalOthers Figure 11: Categorization analysis. The most prevalent error type is commonsense mistakes, accounting for 80% of the errors. These mistakes occur when the model is faced with questions that require commonsense knowledge, such as inter...

    work page 2024