REVIEW 32 cited by
Gated Multimodal Units for Information Fusion
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Gated Multimodal Units for Information Fusion
read the original abstract
This paper presents a novel model for multimodal learning based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. It was evaluated on a multilabel scenario for genre classification of movies using the plot and the poster. The GMU improved the macro f-score performance of single-modality approaches and outperformed other fusion strategies, including mixture of experts models. Along with this work, the MM-IMDb dataset is released which, to the best of our knowledge, is the largest publicly available multimodal dataset for genre prediction on movies.
Forward citations
Cited by 32 Pith papers
-
EpiFormer: Learning Antigen-Antibody Interactions for Epitope Prediction via Geometric Deep Learning
EpiFormer improves epitope prediction F1 score by over 40% via early-fusion cross-attention in GNN layers and sparsity-aware objectives, while recovering known biology as emergent behavior.
-
ConTact: Contact-First Antibody CDR Design via Explicit Interface Reasoning
ConTact introduces a contact-then-act architecture with distance-biased cross-attention and contact-weighted loss for antibody CDR design, reporting 5-6% better backbone RMSD and superior contact metrics on CHIMERA-Be...
-
ConTact: Contact-First Antibody CDR Design via Explicit Interface Reasoning
ConTact decomposes CDR design into surface fingerprint learning, contact prediction, and contact-gated sequence generation using distance-biased attention and weighted loss, reporting 7% RMSD and 10% F1 gains on CHIME...
-
DecepGPT: Schema-Driven Deception Detection with Multicultural Datasets and Robust Multimodal Learning
A new 1695-sample multicultural dataset plus two modules for stable multimodal fusion and modality consistency yield state-of-the-art deception detection with cross-cultural transfer.
-
AeroVerse-SatAgent: UAV-Satellite Collaborative Spatial Reasoning Inspired by the Dual Visual Pathway Theory of Cognitive Neuroscience
SatAgent is a UAV-satellite collaborative spatial reasoning model using geometric 3D encoding, multi-view alignment, and a new 130K dataset that reports 25.91% and 11.69% gains over general and specialized baselines.
-
SkillSpotter: Pose-Aware Multi-View Skilled Action Detection and Grading in Ego-Exo Videos
SkillSpotter raises class-specific mAP from 12.40 to 21.82 and balanced accuracy to 60.40% on Ego-Exo4D by adding adaptive temporal suppression, gated pose fusion, and bidirectional cross-view attention to temporal ac...
-
Learning from Reliable Latent Prompts for Visual Recognition with Missing Modalities
Proposes input-agnostic latent prompts for robust cross-modal compensation in missing-modality visual recognition, claiming SOTA on three benchmarks.
-
Towards Explainable Adjudicative Variance: Quantifying Judicial Discretion via Gated Multi-Task Learning
A Judge-Aware Gated Multi-Task Learning architecture with outcome taxonomy supervision achieves SOTA accuracy on 13,937 UK Employment Tribunal decisions using an order of magnitude fewer parameters than generative SFT...
-
E$^3$C: Video Generation with 3D Environmental Memory and Ego-Exo Human Pose Control
E³C is a video diffusion model that disentangles persistent 3D scene structure via point-cloud memory from human dynamics via ego-exo pose controls for improved egocentric video generation on the Nymeria dataset.
-
MTLLFM: Multimodal-Temporal Laughter Localization: UR-FUNNY-Temporal and SMILE-Temporal Benchmarks with an Adaptive Multimodal Fusion Model
New temporally annotated laughter datasets and a weakly-supervised multimodal model using HuBERT and MAE encoders with adaptive gating achieve 99% F1 and improve downstream reasoning by 227% on CIDEr.
-
AgForce Enables Antigen-conditioned Generative Antibody Design
AgForce improves antigen-conditioned antibody design by using framework dropout, gated bottlenecks, hyperbolic cross attention, MDN sequence head with Potts-like coupling, annealed MCL, and antigen cycle consistency t...
-
EvoStruct: Bridging Evolutionary and Structural Priors for Antibody CDR Design via Protein Language Model Adaptation
EvoStruct integrates evolutionary priors from a protein language model with structural priors from an E(3)-equivariant GNN to raise amino acid recovery by 16% and diversity by 2.3x on CHIMERA-Bench while cutting perpl...
-
Learning Multi-Relational Graph Representations for DNA Methylation-Based Biological Age Estimation
RelAge-GNN models relationships among CpG sites via co-methylation, genomic location, and gene association graphs to estimate biological age more accurately than prior methods.
-
EduGage: Methods and Dataset for Sensor-Based Momentary Assessment of Engagement in Self-Guided Video Learning
EduGage releases a multimodal sensor dataset and models for estimating learner engagement in self-guided video learning, reporting MAE of 0.81 and outperforming baselines with 16 participants.
-
CGCMA: Conditionally-Gated Cross-Modal Attention for Event-Conditioned Asynchronous Fusion
CGCMA separates text-conditioned grounding from lag-aware trust gating to fuse asynchronous price and web data, yielding the highest Sharpe ratio of +0.449 on a new crypto news corpus.
-
Enhancing Foundation VLM Robustness to Missing Modality: Scalable Diffusion for Bi-directional Feature Restoration
A diffusion model with dynamic modality gating and cross-modal mutual learning restores missing features in VLMs bi-directionally while preserving the original model's generalization.
-
Routing-Based Continual Learning for Multimodal Large Language Models
Routing architecture for MLLMs enables continual learning with constant compute, matching multi-task learning performance and supporting cross-modal transfer.
-
Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion
Gated fusion of fastText and BERT embeddings into an end-to-end ASR model captures multi-sentence conversational context and lowers word error rate on the Switchboard corpus.
-
Towards Explainable Adjudicative Variance: Quantifying Judicial Discretion via Gated Multi-Task Learning
A gated multi-task architecture with outcome taxonomy achieves state-of-the-art legal outcome prediction on 13,937 UK Employment Tribunal cases by disentangling judge identity, using far fewer parameters than generati...
-
Missing-Token Prompted Reliability-Aware Fusion for Robust Polyglot Speaker Identification
MRAF framework uses missing-token prompting and reliability-aware cross-attention fusion to achieve 100% accuracy on some POLY-SIM 2026 tasks and competitive results on missing-face cases.
-
I Understand How You Feel: Enhancing Deeper Emotional Support Through Multilingual Emotional Validation in Dialogue System
Releases M-EDESConv and M-TESC multilingual datasets and introduces MEGUMI model that fuses XLM-RoBERTa with emotion encoders for superior validation timing detection.
-
DroneDAR: Long-Range Drone Distance Estimation Using Monocular Vision and Bounding-Box Features
DroneDAR combines a convolutional backbone with bounding-box cues via lightweight gating to predict drone range from image crops and evaluates effects of backbone capacity, resolution, and loss functions across distan...
-
SDM-Q: Cost-Aware Staged Decision-Making for Multi-Omics Classification with Deep Q-Learning
SDM-Q reformulates multi-omics classification as a cost-aware sequential decision process solved with deep Q-learning and reports reduced modality use on four public datasets while preserving accuracy.
-
AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuning
AOEPT proposes modal-contextualized prompts that distill global modality priors to restore reasoning scope in multimodal transformers under missing-modality conditions.
-
Multimodal and Multi-view Models for Emotion Recognition
Multimodal training with attention and contrastive multi-view learning improves both combined and acoustic-only emotion recognition on IEMOCAP over prior acoustic baselines.
-
Gated Multi-Graph Fusion via Graph Attention Networks for Alzheimer's Disease Detection
A gated fusion of graph attention networks on semantic, dependency, and co-occurrence graphs from speech achieves 90% accuracy for Alzheimer's detection on the ADReSSo dataset.
-
Does Role Specialization Matter for Explanation Faithfulness in Mixture-of-Experts?
Representation decorrelation regularization in MoE models improves explanation faithfulness on multimodal benchmarks while preserving task performance.
-
Bridging Classification and Reconstruction: Cooperative Time Series Anomaly Detection
CoAD unifies outlier exposure classification and masked autoencoder reconstruction in a cooperative loop to detect subtle and prolonged time series anomalies.
-
Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities
A two-level reference alignment framework uses complete-modality samples and prototype voting to reduce decision drift and improve robustness in multimodal sentiment analysis under missing modalities.
-
Lightweight Stylistic Consistency Profiling: Robust Detection of LLM-Generated Textual Content for Multimedia Moderation
LiSCP detects LLM-generated text via stylistic consistency profiling across paraphrased variants and reports up to 11.79% better cross-domain accuracy plus robustness to adversarial attacks.
-
Dementia classification from spontaneous speech using wrapper-based feature selection
Using full-recording acoustic features and wrapper selection, the Extreme Minimal Learning Machine provides competitive dementia classification accuracy at lower computational cost on ADReSS and Pitt datasets.
-
A Two-Stage Dual-Modality Model for Facial Emotional Expression Recognition
A dual-modality model combining DINOv2 visual features with Wav2Vec audio features achieves Macro-F1 of 0.5368 on the ABAW validation set for facial expression recognition.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.