Gated Multimodal Units for Information Fusion
read the original abstract
This paper presents a novel model for multimodal learning based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. It was evaluated on a multilabel scenario for genre classification of movies using the plot and the poster. The GMU improved the macro f-score performance of single-modality approaches and outperformed other fusion strategies, including mixture of experts models. Along with this work, the MM-IMDb dataset is released which, to the best of our knowledge, is the largest publicly available multimodal dataset for genre prediction on movies.
This paper has not been read by Pith yet.
Forward citations
Cited by 32 Pith papers
-
EpiFormer: Learning Antigen-Antibody Interactions for Epitope Prediction via Geometric Deep Learning
EpiFormer improves epitope prediction F1 score by over 40% via early-fusion cross-attention in GNN layers and sparsity-aware objectives, while recovering known biology as emergent behavior.
-
ConTact: Contact-First Antibody CDR Design via Explicit Interface Reasoning
ConTact introduces a contact-then-act architecture with distance-biased cross-attention and contact-weighted loss for antibody CDR design, reporting 5-6% better backbone RMSD and superior contact metrics on CHIMERA-Be...
-
ConTact: Contact-First Antibody CDR Design via Explicit Interface Reasoning
ConTact decomposes CDR design into surface fingerprint learning, contact prediction, and contact-gated sequence generation using distance-biased attention and weighted loss, reporting 7% RMSD and 10% F1 gains on CHIME...
-
DecepGPT: Schema-Driven Deception Detection with Multicultural Datasets and Robust Multimodal Learning
A new 1695-sample multicultural dataset plus two modules for stable multimodal fusion and modality consistency yield state-of-the-art deception detection with cross-cultural transfer.
-
AeroVerse-SatAgent: UAV-Satellite Collaborative Spatial Reasoning Inspired by the Dual Visual Pathway Theory of Cognitive Neuroscience
SatAgent is a UAV-satellite collaborative spatial reasoning model using geometric 3D encoding, multi-view alignment, and a new 130K dataset that reports 25.91% and 11.69% gains over general and specialized baselines.
-
SkillSpotter: Pose-Aware Multi-View Skilled Action Detection and Grading in Ego-Exo Videos
SkillSpotter raises class-specific mAP from 12.40 to 21.82 and balanced accuracy to 60.40% on Ego-Exo4D by adding adaptive temporal suppression, gated pose fusion, and bidirectional cross-view attention to temporal ac...
-
Learning from Reliable Latent Prompts for Visual Recognition with Missing Modalities
Proposes input-agnostic latent prompts for robust cross-modal compensation in missing-modality visual recognition, claiming SOTA on three benchmarks.
-
Towards Explainable Adjudicative Variance: Quantifying Judicial Discretion via Gated Multi-Task Learning
A Judge-Aware Gated Multi-Task Learning architecture with outcome taxonomy supervision achieves SOTA accuracy on 13,937 UK Employment Tribunal decisions using an order of magnitude fewer parameters than generative SFT...
-
E$^3$C: Video Generation with 3D Environmental Memory and Ego-Exo Human Pose Control
E³C is a video diffusion model that disentangles persistent 3D scene structure via point-cloud memory from human dynamics via ego-exo pose controls for improved egocentric video generation on the Nymeria dataset.
-
MTLLFM: Multimodal-Temporal Laughter Localization: UR-FUNNY-Temporal and SMILE-Temporal Benchmarks with an Adaptive Multimodal Fusion Model
New temporally annotated laughter datasets and a weakly-supervised multimodal model using HuBERT and MAE encoders with adaptive gating achieve 99% F1 and improve downstream reasoning by 227% on CIDEr.
-
AgForce Enables Antigen-conditioned Generative Antibody Design
AgForce improves antigen-conditioned antibody design by using framework dropout, gated bottlenecks, hyperbolic cross attention, MDN sequence head with Potts-like coupling, annealed MCL, and antigen cycle consistency t...
-
EvoStruct: Bridging Evolutionary and Structural Priors for Antibody CDR Design via Protein Language Model Adaptation
EvoStruct integrates evolutionary priors from a protein language model with structural priors from an E(3)-equivariant GNN to raise amino acid recovery by 16% and diversity by 2.3x on CHIMERA-Bench while cutting perpl...
-
Learning Multi-Relational Graph Representations for DNA Methylation-Based Biological Age Estimation
RelAge-GNN models relationships among CpG sites via co-methylation, genomic location, and gene association graphs to estimate biological age more accurately than prior methods.
-
EduGage: Methods and Dataset for Sensor-Based Momentary Assessment of Engagement in Self-Guided Video Learning
EduGage releases a multimodal sensor dataset and models for estimating learner engagement in self-guided video learning, reporting MAE of 0.81 and outperforming baselines with 16 participants.
-
CGCMA: Conditionally-Gated Cross-Modal Attention for Event-Conditioned Asynchronous Fusion
CGCMA separates text-conditioned grounding from lag-aware trust gating to fuse asynchronous price and web data, yielding the highest Sharpe ratio of +0.449 on a new crypto news corpus.
-
Enhancing Foundation VLM Robustness to Missing Modality: Scalable Diffusion for Bi-directional Feature Restoration
A diffusion model with dynamic modality gating and cross-modal mutual learning restores missing features in VLMs bi-directionally while preserving the original model's generalization.
-
Routing-Based Continual Learning for Multimodal Large Language Models
Routing architecture for MLLMs enables continual learning with constant compute, matching multi-task learning performance and supporting cross-modal transfer.
-
Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion
Gated fusion of fastText and BERT embeddings into an end-to-end ASR model captures multi-sentence conversational context and lowers word error rate on the Switchboard corpus.
-
Towards Explainable Adjudicative Variance: Quantifying Judicial Discretion via Gated Multi-Task Learning
A gated multi-task architecture with outcome taxonomy achieves state-of-the-art legal outcome prediction on 13,937 UK Employment Tribunal cases by disentangling judge identity, using far fewer parameters than generati...
-
Missing-Token Prompted Reliability-Aware Fusion for Robust Polyglot Speaker Identification
MRAF framework uses missing-token prompting and reliability-aware cross-attention fusion to achieve 100% accuracy on some POLY-SIM 2026 tasks and competitive results on missing-face cases.
-
I Understand How You Feel: Enhancing Deeper Emotional Support Through Multilingual Emotional Validation in Dialogue System
Releases M-EDESConv and M-TESC multilingual datasets and introduces MEGUMI model that fuses XLM-RoBERTa with emotion encoders for superior validation timing detection.
-
DroneDAR: Long-Range Drone Distance Estimation Using Monocular Vision and Bounding-Box Features
DroneDAR combines a convolutional backbone with bounding-box cues via lightweight gating to predict drone range from image crops and evaluates effects of backbone capacity, resolution, and loss functions across distan...
-
SDM-Q: Cost-Aware Staged Decision-Making for Multi-Omics Classification with Deep Q-Learning
SDM-Q reformulates multi-omics classification as a cost-aware sequential decision process solved with deep Q-learning and reports reduced modality use on four public datasets while preserving accuracy.
-
AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuning
AOEPT proposes modal-contextualized prompts that distill global modality priors to restore reasoning scope in multimodal transformers under missing-modality conditions.
-
Multimodal and Multi-view Models for Emotion Recognition
Multimodal training with attention and contrastive multi-view learning improves both combined and acoustic-only emotion recognition on IEMOCAP over prior acoustic baselines.
-
Gated Multi-Graph Fusion via Graph Attention Networks for Alzheimer's Disease Detection
A gated fusion of graph attention networks on semantic, dependency, and co-occurrence graphs from speech achieves 90% accuracy for Alzheimer's detection on the ADReSSo dataset.
-
Does Role Specialization Matter for Explanation Faithfulness in Mixture-of-Experts?
Representation decorrelation regularization in MoE models improves explanation faithfulness on multimodal benchmarks while preserving task performance.
-
Bridging Classification and Reconstruction: Cooperative Time Series Anomaly Detection
CoAD unifies outlier exposure classification and masked autoencoder reconstruction in a cooperative loop to detect subtle and prolonged time series anomalies.
-
Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities
A two-level reference alignment framework uses complete-modality samples and prototype voting to reduce decision drift and improve robustness in multimodal sentiment analysis under missing modalities.
-
Lightweight Stylistic Consistency Profiling: Robust Detection of LLM-Generated Textual Content for Multimedia Moderation
LiSCP detects LLM-generated text via stylistic consistency profiling across paraphrased variants and reports up to 11.79% better cross-domain accuracy plus robustness to adversarial attacks.
-
Dementia classification from spontaneous speech using wrapper-based feature selection
Using full-recording acoustic features and wrapper selection, the Extreme Minimal Learning Machine provides competitive dementia classification accuracy at lower computational cost on ADReSS and Pitt datasets.
-
A Two-Stage Dual-Modality Model for Facial Emotional Expression Recognition
A dual-modality model combining DINOv2 visual features with Wav2Vec audio features achieves Macro-F1 of 0.5368 on the ABAW validation set for facial expression recognition.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.