LRS3-TED: a large-scale dataset for visual speech recognition
read the original abstract
This paper introduces a new multi-modal dataset for visual and audio-visual speech recognition. It includes face tracks from over 400 hours of TED and TEDx videos, along with the corresponding subtitles and word alignment boundaries. The new dataset is substantially larger in scale compared to other public datasets that are available for general research.
This paper has not been read by Pith yet.
Forward citations
Cited by 19 Pith papers
-
HoliDubber: Holistic Video Dubbing for Complex Acoustic Scenes via Text-Guided Audio Synthesis
HoliDubber introduces a patch-based autoregressive diffusion transformer for joint text-guided synthesis of speech and ambient audio in video dubbing, with a new benchmark showing outperformance over prior speech-only...
-
Hierarchical Codec Diffusion for Video-to-Speech Generation
HiCoDiT generates speech from video by conditioning low-level RVQ tokens on speaker identity and high-level tokens on facial expressions via a dual-scale normalized diffusion transformer.
-
CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing
CoSyncDiT is a cognitive-inspired diffusion transformer that achieves state-of-the-art lip synchronization and naturalness in movie dubbing by guiding noise-to-speech generation through acoustic, visual, and contextua...
-
OmniSonic: Towards Universal and Holistic Audio Generation from Video and Text
OmniSonic introduces a TriAttn-DiT architecture with MoE gating to jointly generate on-screen, off-screen, and speech audio from video and text, outperforming prior models on a new UniHAGen-Bench.
-
Audio-visual Contrastive Alignment for Diffusion-based Visual-conditioned Speech Enhancement
Augmenting diffusion-based visual-conditioned speech enhancement with a contrastive audio-visual loss produces consistent gains in interference suppression and perceptual quality, especially at low SNRs.
-
The Lipreading Gap: Do VSR Models Perceive Visual Speech Like Human Lipreaders?
Empirical comparison on MaFI dataset shows VSR models outperform humans overall but correlate with linguistic priors over visual features, unlike human lipreaders.
-
Assessing True Generalisability of Audio-Visual Speech Recognisers
State-of-the-art AVSR models show substantial performance degradation on a new distribution-matched test set from MultiVSR, indicating failure to generalise beyond the LRS3 benchmark.
-
M2S-AVSR: Modality-aware Multi-view Self-supervised Representation for Robust Audio-Visual Speech Recognition
M2S-AVSR introduces multi-view self-supervised visual encoding and modality-aware fusion for AVSR, releases the AISHELL8-RealScene dataset, and reports relative gains up to 29.4% on LRS3 under perturbations plus new S...
-
LRS-VoxMM: A benchmark for in-the-wild audio-visual speech recognition
LRS-VoxMM is a new in-the-wild AVSR benchmark that is harder than LRS3 and demonstrates increasing value of visual information under acoustic degradation.
-
Tracking Listener Attention: Gaze-Guided Audio-Visual Speech Enhancement Framework
The GG-AVSE framework uses listener gaze direction combined with YOLO5Face and AVSEMamba to resolve target-speaker ambiguity in audio-visual speech enhancement, yielding gains in PESQ, STOI, and SI-SDR.
-
Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesis
Experiments with a video-text-to-speech transformer show co-temporal positional indexing enables synchronization without timestamps, text and video supply complementary signals, and modality ordering creates a trade-o...
-
Orthogonal Subspace Decomposition for Generalizable AI-Generated Image Detection
Orthogonal subspace decomposition via SVD on vision foundation model features preserves high-rank pre-trained knowledge by freezing principal components and adapting residuals, reducing overfitting for better generali...
-
My lips are concealed: Audio-visual speech enhancement through obstructions
A deep audio-visual speech enhancement network separates a speaker's voice using lip movements and voice representations, with self-enrollment to handle visual occlusions.
-
CineDance: Towards Next-Generation Multi-Shot Long-Form Cinematic Audio-Video Generation
Introduces CineDance-1M dataset for multi-shot long-form text-to-audio-video generation along with CineBench and a model adaptation.
-
Head-Pose-Aware Visual Speech Recognition with FiLM Modulation
HP-VSR-ResFiLM adds a single residual FiLM modulation block conditioned on head pose to a CNN visual encoder, yielding WER of 25.0% on LRS2 and 33.2% on LRS3 under standard training conditions.
-
HighSync: High-Quality Lip Synchronization via Latent Diffusion Models
HighSync is a diffusion-based lip synchronization system that operates natively at 512x512 resolution by eliminating data leakage to enforce genuine audio dependence and reports state-of-the-art results on quality and...
-
Delayed Commitment for Representation Readiness in Stage-wise Audio-Visual Learning
DPC-Net improves stage-wise audio-visual learning by correcting readiness deficiencies in fused representations using cross-layer and cross-modal evidence.
-
Interpreting the Role of Visemes in Audio-Visual Speech Recognition
Interpretability analysis of AV-HuBERT reveals visual-driven clustering of visemes that audio refines, especially for ambiguous cases.
-
BUT System Description for CHiME-9 MCoRec Challenge
BUT's CHiME-9 MCoRec system conditions Parakeet-v2 ASR on AV-HuBERT visuals for 33.7% WER and uses Qwen3.5 LLM for hierarchical clustering to reach 0.97 F1, beating the baseline by 16.2% WER and 0.15 F1 on the develop...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.