pith. sign in

arxiv: 2210.13438 · v1 · pith:ZFMZGE3Cnew · submitted 2022-10-24 · 📡 eess.AS · cs.AI· cs.SD· stat.ML

High Fidelity Neural Audio Compression

Alexandre D\'efossez , Jade Copet , Gabriel Synnaeve , Yossi Adi This is my paper

Pith reviewed 2026-05-13 21:46 UTC · model grok-4.3

classification 📡 eess.AS cs.AIcs.SDstat.ML
keywords neural audio codechigh-fidelity compressionstreaming encoder-decoderquantized latent spacemultiscale spectrogram adversaryloss balancertransformer compressionMUSHRA evaluation
0
0 comments X

The pith

A neural network audio codec with streaming encoder-decoder and quantized latents delivers higher fidelity than baselines at real-time speeds.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a neural audio codec that compresses speech and music while keeping perceptual quality high enough to beat prior methods in listening tests. It trains an end-to-end streaming encoder-decoder whose latent space is quantized, using one multiscale spectrogram discriminator to suppress artifacts and a loss balancer that treats each loss weight as the target fraction of the total gradient. The same architecture supports optional lightweight Transformer layers that cut the final bitrate by up to 40 percent without losing real-time performance. These choices matter because real-time, high-quality compression is required for efficient streaming, storage, and transmission of audio on bandwidth-constrained devices. The authors show the gains hold for 24 kHz monophonic and 48 kHz stereophonic signals across clean speech, noisy-reverberant speech, and music.

Core claim

The authors present a real-time high-fidelity neural audio codec built from a streaming encoder-decoder with a quantized latent space trained end-to-end. Training is stabilized by a single multiscale spectrogram adversary that reduces artifacts and by a novel loss-balancer module in which each loss weight directly sets the fraction of the overall gradient it contributes. Lightweight Transformer models can be stacked on the quantized representation to achieve up to 40 percent additional compression while remaining faster than real time. MUSHRA subjective tests across multiple bandwidths and domains establish superiority over existing codecs for both 24 kHz monophonic and 48 kHz stereophonic音频

What carries the argument

Streaming encoder-decoder architecture with quantized latent space, trained using a single multiscale spectrogram adversary and a loss-balancer mechanism that decouples loss weights from gradient scale.

Load-bearing premise

The MUSHRA listening tests on the chosen audio domains and bandwidths are representative of real-world use and the model does not overfit to the training distribution in ways that degrade on unseen content.

What would settle it

A new MUSHRA test on audio outside the training domains (for example live concert recordings or rare speech accents) showing the neural codec no longer rated higher than the baselines at the same bitrate.

read the original abstract

We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary that efficiently reduces artifacts and produce high-quality samples. We introduce a novel loss balancer mechanism to stabilize training: the weight of a loss now defines the fraction of the overall gradient it should represent, thus decoupling the choice of this hyper-parameter from the typical scale of the loss. Finally, we study how lightweight Transformer models can be used to further compress the obtained representation by up to 40%, while staying faster than real time. We provide a detailed description of the key design choices of the proposed model including: training objective, architectural changes and a study of various perceptual loss functions. We present an extensive subjective evaluation (MUSHRA tests) together with an ablation study for a range of bandwidths and audio domains, including speech, noisy-reverberant speech, and music. Our approach is superior to the baselines methods across all evaluated settings, considering both 24 kHz monophonic and 48 kHz stereophonic audio. Code and models are available at github.com/facebookresearch/encodec.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 3 minor

Summary. The paper introduces EnCodec, a real-time neural audio codec using a streaming encoder-decoder with quantized latent representations trained end-to-end. Key contributions include a single multiscale spectrogram discriminator to reduce artifacts, a novel loss balancer that sets loss weights as target gradient fractions to stabilize training, and optional lightweight Transformer models for up to 40% further compression of the latent codes. The work reports extensive MUSHRA subjective evaluations across speech, noisy-reverberant speech, and music at multiple bandwidths for both 24 kHz mono and 48 kHz stereo audio, claiming consistent superiority over published baselines, with code and models released for reproducibility.

Significance. If the reported MUSHRA rankings hold, the work provides a practical advance in high-fidelity, low-latency neural audio compression with direct applicability to streaming and storage. Strengths include the public release of code and models, detailed ablation studies, and the loss-balancer formulation that decouples hyper-parameter choice from loss scale; these elements support replication and extension beyond the specific domains tested.

minor comments (3)
  1. [§3.3] §3.3: The single multiscale spectrogram discriminator is described at a high level; adding the exact frequency scales and window sizes used would aid exact replication.
  2. [Table 2] Table 2: The MUSHRA scores for the 48 kHz stereo music condition would benefit from reported confidence intervals or standard deviations to quantify variability across listeners.
  3. [§4.2] §4.2: The claim of 'parameter-free' behavior for certain loss terms is not fully supported by the listed free parameters (number of residual codebooks and balancer targets); a brief clarification on which quantities remain fixed would improve precision.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive summary, recognition of the practical contributions (including the loss balancer, public code release, and MUSHRA evaluations), and the recommendation to accept. We are pleased that the work's applicability to streaming audio was noted.

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper presents an empirical neural audio codec (encoder-decoder with quantization, single multiscale spectrogram discriminator, and a proposed loss-balancer that normalizes gradient contributions by design). All central claims of superiority are grounded in external MUSHRA listening tests across speech, music, and stereo domains plus comparisons to published baselines, with code released for replication. No equation or training step reduces by construction to a fitted parameter renamed as a prediction, no self-citation chain is load-bearing for the architecture or results, and the loss-balancer is introduced as an explicit mechanism rather than derived from the target metric. The derivation chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 0 invented entities

The model relies on standard assumptions of end-to-end differentiable training and perceptual loss functions; no new physical entities are postulated. Free parameters include the number of codebooks, codebook size, and the target gradient fractions in the loss balancer, all chosen by hand or grid search.

free parameters (2)
  • number of residual codebooks
    Chosen to achieve target bitrate; directly controls the discrete representation size.
  • loss balancer target fractions
    Hyper-parameters that set the desired gradient contribution of each loss term; fitted to stabilize training.
axioms (1)
  • domain assumption Multiscale spectrogram discrimination is sufficient to suppress perceptual artifacts in audio reconstruction.
    Invoked in the training objective section to justify using a single discriminator instead of multiple.

pith-pipeline@v0.9.0 · 5540 in / 1318 out tokens · 33298 ms · 2026-05-13T21:46:14.260988+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. WavTTS: Towards High-Quality Zero-Shot TTS via Direct Raw Waveform Modeling

    eess.AS 2026-06 unverdicted novelty 8.0

    WavTTS is the first raw-waveform diffusion TTS model using DiT flow matching and multi-scale mel supervision that approaches SOTA latent zero-shot performance while beating prior end-to-end models.

  2. MusicLM: Generating Music From Text

    cs.SD 2023-01 conditional novelty 8.0

    MusicLM produces coherent multi-minute 24 kHz music from text prompts using hierarchical sequence-to-sequence modeling and outperforms prior systems in quality and text adherence.

  3. HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures

    cs.LG 2026-07 unverdicted novelty 7.0

    HERMES provides a reusable hierarchical labeling substrate for pre-training data that reveals granularity-specific effects in data mixing rules during model training.

  4. FlexiSLM: A Dynamic and Controllable Frame Rate Spoken Language Model

    cs.SD 2026-06 unverdicted novelty 7.0

    FlexiSLM is the first spoken language model supporting dynamic and controllable frame rates on speech input and output, outperforming fixed-rate 7B models at high quality and enabling faster inference at lower rates l...

  5. DTM-Codec: Dynamic Token Masking for VFR Speech Coding with Efficient Boundary Selection

    eess.AS 2026-06 unverdicted novelty 7.0

    DTM-Codec achieves better reconstruction quality and intelligibility than fixed-frame-rate neural speech codecs at matched total bitrate via dynamic token masking and Path Length Equalization for variable frame rates.

  6. AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation

    eess.AS 2026-06 unverdicted novelty 7.0

    AudioCALM presents a continuous autoregressive framework with flow-matching prediction and A-MoME architecture that unifies speech, sound, and music generation while matching modality-specific state-of-the-art performance.

  7. NAC: Neural Action Codec for Vision-Language-Action Models

    cs.RO 2026-06 unverdicted novelty 7.0

    NAC adapts multi-scale RVQGAN audio codecs with kinematic-specific losses to produce ordered action tokens that yield lower reconstruction error and higher task success than prior tokenizers in VLA models.

  8. A Survey of Full-Duplex Spoken Dialogue Systems: Architectural Hierarchy, Interaction Ontology, and Decision State Machine

    eess.AS 2026-06 accept novelty 7.0

    A survey proposing an L0-L3 architectural hierarchy, T×I×R interaction ontology, and IDLE/LISTEN/SPEAK/WAIT/DUAL decision state machine for full-duplex spoken dialogue systems, documenting a realization gap between ar...

  9. HoliDubber: Holistic Video Dubbing for Complex Acoustic Scenes via Text-Guided Audio Synthesis

    eess.AS 2026-06 unverdicted novelty 7.0

    HoliDubber introduces a patch-based autoregressive diffusion transformer for joint text-guided synthesis of speech and ambient audio in video dubbing, with a new benchmark showing outperformance over prior speech-only...

  10. Probing Spatial Structure in Pretrained Audio Representations

    cs.SD 2026-06 unverdicted novelty 7.0

    Introduces SARL benchmark showing pretrained audio encoders encode source-level spatial factors more readily than room-level factors, with patterns shaped by input configuration and training paradigm.

  11. How Neural Losses Shape VAE Latents

    cs.LG 2026-05 unverdicted novelty 7.0

    Neural reconstruction losses in VAEs reduce latent information content and produce more isotropic latent geometries with even uncertainty distribution.

  12. Prompt Codebooks: Discrete Compositional Optimization for Language Model Instruction Refinement

    cs.AI 2026-05 unverdicted novelty 7.0

    Prompt Codebooks recasts automatic prompt optimization as discrete learning over a finite vocabulary of atomic natural-language instincts with per-instance routing, yielding up to +30.36 point gains over zero-shot and...

  13. Academic Text-to-Music Grand Challenge: Datasets, Baselines, and Evaluation Methods

    cs.SD 2026-05 unverdicted novelty 7.0

    Presents the ATTM grand challenge with efficiency and performance tracks for text-to-music generation using a public instrumental music dataset, evaluated via FAD, CLAP, a new CCS metric, and subjective tests.

  14. Codec-Robust Attacks on Audio LLMs

    cs.SD 2026-05 unverdicted novelty 7.0

    CodecAttack perturbs audio in codec latent space with multi-bitrate EoT to achieve 85.5% average ASR on Opus-compressed Audio LLMs versus under 26% for waveform baselines, with transfer to MP3 and AAC.

  15. Modeling Music as a Time-Frequency Image: A 2D Tokenizer for Music Generation

    cs.SD 2026-05 unverdicted novelty 7.0

    BandTok tokenizes Mel-spectrograms as independent time-frequency band tokens from a single codebook and pairs it with 2D RoPE in an autoregressive model to improve music generation over residual multi-codebook tokenizers.

  16. AffectCodec: Emotion-Preserving Neural Speech Codec for Expressive Speech Modeling

    cs.SD 2026-05 unverdicted novelty 7.0

    AffectCodec is an emotion-guided neural speech codec that preserves emotional cues during quantization while maintaining semantic fidelity and prosodic naturalness.

  17. VITA-QinYu: Expressive Spoken Language Model for Role-Playing and Singing

    cs.CL 2026-05 unverdicted novelty 7.0

    VITA-QinYu is the first expressive end-to-end spoken language model supporting role-playing and singing alongside conversation, trained on 15.8K hours of data and outperforming prior models on expressiveness and conve...

  18. PairAlign: A Framework for Sequence Tokenization via Self-Alignment with Applications to Audio Tokenization

    cs.LG 2026-05 unverdicted novelty 7.0

    PairAlign learns compact audio token sequences via self-alignment of paired content views using an autoregressive decoder, achieving strong cross-view consistency and edit-distance preservation while reducing token co...

  19. PairAlign: A Framework for Sequence Tokenization via Self-Alignment with Applications to Audio Tokenization

    cs.LG 2026-05 unverdicted novelty 7.0

    PairAlign learns compact variable-length token sequences for audio via self-alignment on paired content-preserving views, achieving 55% fewer archive tokens than VQ while preserving edit-distance retrieval at 12.71 tokens/s.

  20. Indic-CodecFake meets SATYAM: Towards Detecting Neural Audio Codec Synthesized Speech Deepfakes in Indic Languages

    eess.AS 2026-04 unverdicted novelty 7.0

    Introduces the Indic-CodecFake dataset for Indic codec deepfakes and SATYAM, a novel hyperbolic ALM that outperforms baselines through dual-stage semantic-prosodic fusion using Bhattacharya distance.

  21. When Spoof Detectors Travel: Evaluation Across 66 Languages in the Low-Resource Language Spoofing Corpus

    cs.SD 2026-03 unverdicted novelty 7.0

    LRLspoof corpus and threshold-transfer evaluation demonstrate that spoof detection performance varies markedly across languages, identifying language as an independent domain shift factor.

  22. Steering Autoregressive Music Generation with Recursive Feature Machines

    cs.LG 2025-10 unverdicted novelty 7.0

    MusicRFM discovers interpretable concept directions in music model hidden states using RFM probes and injects them at inference to steer generation toward desired musical properties without retraining.

  23. Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation

    cs.CV 2023-10 unverdicted novelty 7.0

    A new shared video-image tokenizer enables large language models to surpass diffusion models on standard visual generation benchmarks.

  24. Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

    cs.CL 2023-01 unverdicted novelty 7.0

    VALL-E is a neural codec language model trained on 60K hours of speech that performs zero-shot TTS, synthesizing natural speech that matches an unseen speaker's voice, emotion, and environment from a 3-second prompt.

  25. Learning to Evade: Adaptive Attacks on Audio Watermarking

    cs.SD 2026-06 unverdicted novelty 6.0

    Introduces AWM adaptive attack using two-stage optimization and distribution estimation to bypass audio watermark detectors with low detection rates on voice datasets.

  26. Bridging the Age Gap: Towards Detecting Neural Audio Codec Synthesized Elderly Speech Deepfake

    eess.AS 2026-06 unverdicted novelty 6.0

    Defines ECFD task, releases ECF dataset, demonstrates poor generalization of prior detectors to elderly speech, and introduces BONSAI fusion of LanguageBind and ImageBind achieving 1.66% average EER.

  27. Towards Detecting Neural Audio Codec Synthesized Heart Sounds

    eess.AS 2026-06 unverdicted novelty 6.0

    Introduces SHAC task, CARDIOFAKE dataset, and GROOT fusion model achieving SOTA detection of neural codec-synthesized phonocardiograms using MFCC and WavLM features.

  28. DisSpeech: Low-Resource Controllable Mandarin Stuttered Speech Synthesis for ASR Augmentation

    cs.SD 2026-06 unverdicted novelty 6.0

    DisSpeech synthesizes controllable stuttered Mandarin speech via discrete tokens and stuttering event labels to augment ASR datasets, improving recognition to 4.19% CER on stuttered tasks with minimal impact on fluent speech.

  29. Exploiting Neural Audio Codec Latents for Adversarial Audio Attacks

    cs.SD 2026-06 unverdicted novelty 6.0

    A conditional generator operating in neural audio codec latent space produces targeted adversarial audio examples in one forward pass, reaching up to 99% success rate at sub-7 ms inference.

  30. Self-Guidance: Enhancing Neural Codecs via Decoder Manifold Alignment

    cs.SD 2026-06 unverdicted novelty 6.0

    Self-guidance adds a lightweight feature-mapping loss to align decoder manifolds in VQ-VAE speech codecs, raising reconstruction metrics and allowing 4x codebook reduction with no fidelity loss.

  31. TLDR: Compressing Audio Tokens for Efficient Autoregressive Text-to-Speech

    cs.SD 2026-06 unverdicted novelty 6.0

    TLDR groups codec tokens into patches for patch-level autoregressive modeling in pretrained TTS systems, yielding 1.8x speedup and 75% KV-cache reduction at patch size 4.

  32. Continuous Audio Thinking for Large Audio Language Models

    cs.CL 2026-06 unverdicted novelty 6.0

    CoAT adds a continuous latent thinking space to LALMs via expert distillation to retain acoustic information, yielding gains on audio reasoning, understanding, music, emotion, and transcription benchmarks across three models.

  33. JenBridge: Adaptive Long-Form Video Soundtracking across Scene Transitions

    cs.SD 2026-06 unverdicted novelty 6.0

    JenBridge pretrains a flow-matching Transformer on text-audio data then adapts it with video conditioning and an LLM director to select transitions, claiming better coherence than prior methods on a new LVS benchmark.

  34. UniVocal: Unified Speech-Singing Code-Switching Synthesis

    cs.SD 2026-06 unverdicted novelty 6.0

    UniVocal presents a text-context-only framework for speech-singing code-switching synthesis via two-stage curriculum learning and a synthetic data pipeline, claiming SOTA on a new benchmark.

  35. AnchorSteer: Self-Discovered Concept Injection for Structure-Preserving Music Editing

    cs.SD 2026-05 unverdicted novelty 6.0

    AnchorSteer couples self-discovered semantic concept vectors with structural anchoring in diffusion models to achieve controllable music editing with preserved structure.

  36. Audio Pirates: Black-box Audio Watermark Removal via Diffusion Priors

    cs.CR 2026-05 unverdicted novelty 6.0

    DiffErase removes black-box audio watermarks via diffusion priors by adding intermediate noise and regenerating with a pretrained model, preserving quality across audio domains.

  37. Codec-Robust Attacks on Audio LLMs

    cs.SD 2026-05 unverdicted novelty 6.0

    CodecAttack optimizes perturbations in neural audio codec latent space to reach 85.5% average target-substring ASR on compressed Opus audio while waveform baselines stay below 26%.

  38. Taming Audio VAEs via Target-KL Regularization

    cs.SD 2026-05 unverdicted novelty 6.0

    The paper introduces target-KL regularization to train audio VAEs at specific bitrates, enabling rate-distortion curves and comparison to discrete audio codecs for improved text-to-sound generation.

  39. A General Differentiable Ray-Wave Framework for Hybrid Refractive-Diffractive System Modeling and Optimization

    physics.optics 2026-05 unverdicted novelty 6.0

    A plug-and-play differentiable model bridging ray and wave optics for hybrid systems that enables end-to-end optimization of planar and conformal diffractive elements.

  40. Seconds-Aligned PCA-DAC Latent Diffusion for Symbolic-to-Audio Drum Rendering

    cs.SD 2026-05 unverdicted novelty 6.0

    Sec2Drum-DAC renders drum audio from symbolic inputs via diffusion on PCA-reduced DAC latents, improving spectral and transient metrics over regression baselines on 1733 held-out windows.

  41. Compact Latent Manifold Translation: A Parameter-Efficient Foundation Model for Cross-Modal and Cross-Frequency Physiological Signal Synthesis

    eess.SP 2026-05 unverdicted novelty 6.0

    A compact 0.09B model using hierarchical discrete tokenization and prompted latent translation outperforms larger baselines in cross-modal PPG-to-ECG synthesis and cross-frequency super-resolution.

  42. MiniMind-O Technical Report: An Open Small-Scale Speech-Native Omni Model

    cs.SD 2026-05 accept novelty 6.0

    MiniMind-O delivers a working 0.1B-scale open omni model with speech-native output, Thinker-Talker split, frozen encoders, and full release of code, checkpoints, and training data.

  43. Text-To-Speech with Chain-of-Details: modeling temporal dynamics in speech generation

    eess.AS 2026-04 unverdicted novelty 6.0

    Chain-of-Details (CoD) is a cascaded TTS method that explicitly models temporal coarse-to-fine dynamics with a shared decoder, achieving competitive performance using significantly fewer parameters.

  44. LLM-Codec: Neural Audio Codec Meets Language Model Objectives

    cs.SD 2026-04 unverdicted novelty 6.0

    LLM-Codec augments audio codec training with multi-step token prediction and contrastive semantic alignment to improve both waveform reconstruction and autoregressive predictability for speech language models.

  45. HCFD: A Benchmark for Audio Deepfake Detection in Healthcare

    eess.AS 2026-04 unverdicted novelty 6.0

    HCFD is a new pathology-aware benchmark and dataset for codec-fake audio detection in healthcare, with PHOENIX-Mamba achieving up to 97% accuracy by modeling fakes as modes in hyperbolic space.

  46. Efficient Training for Cross-lingual Speech Language Models

    cs.CL 2026-04 unverdicted novelty 6.0

    CSLM achieves cross-modal and cross-lingual alignment in speech LLMs via continual pre-training on discrete tokens and speech-text interleaved instruction tuning, enabling scalability without massive speech datasets.

  47. Discrete Token Modeling for Multi-Stem Music Source Separation with Language Models

    eess.AS 2026-04 unverdicted novelty 6.0

    A Conformer-conditioned decoder-only language model generates discrete tokens via a neural audio codec to separate four music stems, reaching near state-of-the-art perceptual quality and top NISQA on vocals in MUSDB18...

  48. Diff-VS: Efficient Audio-Aware Diffusion U-Net for Vocals Separation

    eess.AS 2026-04 unverdicted novelty 6.0

    Diff-VS is an efficient audio-aware diffusion U-Net for vocal separation that matches discriminative baselines on objective metrics while achieving state-of-the-art perceptual quality via proxy measures.

  49. Qwen3-TTS Technical Report

    cs.SD 2026-01 unverdicted novelty 6.0

    Qwen3-TTS delivers state-of-the-art multilingual TTS performance with 3-second voice cloning, description control, and ultra-low-latency streaming via dual tokenizers and a dual-track LM architecture trained on over 5...

  50. Two-Dimensional Quantization for Geometry-Aware Audio Coding

    cs.SD 2025-12 unverdicted novelty 6.0

    Q2D2 uses 2D geometric grid projections to quantize feature pairs in neural audio codecs, yielding implicit codebooks that improve efficiency and utilization over RVQ, VQ, and FSQ while maintaining reconstruction quality.

  51. HarmonicAttack: An Adaptive Cross-Domain Audio Watermark Removal

    cs.SD 2025-11 conditional novelty 6.0

    A black-box audio watermark removal attack trained on limited samples that generalizes across datasets and watermark schemes with high attack success rates.

  52. CodecSep: Prompt-Driven Universal Sound Separation on Neural Audio Codec Latents

    cs.SD 2025-09 unverdicted novelty 6.0

    CodecSep performs prompt-driven universal sound separation directly in neural audio codec latents by combining a frozen DAC backbone with a lightweight FiLM-conditioned Transformer masker driven by CLAP embeddings, yi...

  53. Step-Audio 2 Technical Report

    cs.CL 2025-07 unverdicted novelty 6.0

    Step-Audio 2 integrates a latent audio encoder, reasoning-centric reinforcement learning, and discrete audio token generation into language modeling to deliver state-of-the-art performance on audio understanding and c...

  54. Perceptual implications of automatic anonymization in pathological speech

    eess.AS 2025-05 conditional novelty 6.0

    Listeners detect automatic anonymization in pathological speech at 91-93% accuracy with a 30-point perceived quality drop, yet clinical severity ratings stay nearly unchanged for dysarthria, dysglossia, and dysphonia.

  55. Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound

    cs.SD 2025-02 unverdicted novelty 6.0

    Unified no-reference models assess audio aesthetics across speech, music, and sound via four perceptual axes and achieve performance comparable or superior to human mean opinion scores.

  56. CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation

    cs.RO 2024-11 unverdicted novelty 6.0

    CogACT is a new VLA model that uses a conditioned diffusion action transformer to achieve over 35% higher average success rates than OpenVLA in simulation and 55% in real-robot experiments while generalizing to new ro...

  57. Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesis

    cs.MM 2024-11 unverdicted novelty 6.0

    Experiments with a video-text-to-speech transformer show co-temporal positional indexing enables synchronization without timestamps, text and video supply complementary signals, and modality ordering creates a trade-o...

  58. Better & Faster Large Language Models via Multi-token Prediction

    cs.CL 2024-04 conditional novelty 6.0

    Multi-token prediction training yields higher sample efficiency, better benchmark scores on code generation, and up to 3x faster inference than standard next-token prediction for LLMs.

  59. Compact 3D Gaussian Splatting For Dense Visual SLAM

    cs.CV 2024-03 unverdicted novelty 6.0

    A compact 3D Gaussian Splatting SLAM system reduces Gaussian count and parameter size via masking and a geometry codebook while preserving SOTA reconstruction quality and pose accuracy.

  60. Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models

    eess.AS 2023-11 unverdicted novelty 6.0

    Qwen-Audio trains a unified model on diverse audio and tasks with hierarchical tags to enable strong zero-shot performance on audio understanding benchmarks and multi-turn audio chat.

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages · cited by 88 Pith papers · 5 internal anchors

  1. [1]

    Wv-mos: Mos score prediction by fine-tuned wav2vec 2.0.arXiv preprint arXiv:2203.13086,

    Pavel Andreev, Aibek Alanov, Oleg Ivanov, and Dmitry Vetrov. Hifi++: a unified framework for neural vocoding, bandwidth extension and speech enhancement.arXiv preprint arXiv:2203.13086 ,

    work page arXiv

  2. [2]

    Common voice: A massively-multilingual speech corpus,

    Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670 ,

    work page arXiv 1912

  3. [3]

    Layer Normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

    Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation.arXiv preprint arXiv:1308.3432 ,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Shlomo E Chazan, Lior Wolf, Eliya Nachmani, and Yossi Adi

    URL http: //hdl.handle.net/10230/42015. Shlomo E Chazan, Lior Wolf, Eliya Nachmani, and Yossi Adi. Single channel voice separation for unknown number of speakers under reverberant and noisy settings. InICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 3730–3734. IEEE,

    work page 2021

  6. [6]

    Global - 2021 forecast highlights - cisco

    Cisco. Global - 2021 forecast highlights - cisco. https://www.cisco.com/c/dam/m/en_us/solutions/ service-provider/vni-forecast-highlights/pdf/Global_2021_Forecast_Highlights.pdf,

    work page 2021

  7. [7]

    Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

    Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus).arXiv preprint arXiv:1511.07289 ,

    work page Pith review arXiv

  8. [8]

    Music source separation in the waveform domain,

    Alexandre Défossez, Nicolas Usunier, Léon Bottou, and Francis Bach. Music source separation in the waveform domain. arXiv preprint arXiv:1911.13254 ,

    work page arXiv 1911

  9. [9]

    Real time speech enhancement in the waveform domain.arXiv preprint arXiv:2006.12847,

    Alexandre Defossez, Gabriel Synnaeve, and Yossi Adi. Real time speech enhancement in the waveform domain. arXiv preprint arXiv:2006.12847 ,

    work page arXiv 2006

  10. [10]

    arXiv preprint arXiv:2104.09987 , year=

    Alexandre Défossez, Yossi Adi, and Gabriel Synnaeve. Differentiable model compression via pseudo quantiza- tion noise. arXiv preprint arXiv:2104.09987 ,

    work page arXiv

  11. [11]

    Jukebox: A Generative Model for Music

    Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music.arXiv preprint arXiv:2005.00341 ,

    work page Pith review arXiv 2005

  12. [12]

    Icassp 2022 deep noise suppression challenge

    Harishchandra Dubey, Vishak Gopal, Ross Cutler, Sergiy Matusevych, Sebastian Braun, Emre Sefik Eskimez, Manthan Thakker, Takuya Yoshioka, Hannes Gamper, and Robert Aichner. Icassp 2022 deep noise suppression challenge. InICASSP,

    work page 2022

  13. [13]

    Low bit-rate speech coding with vq-vae and a wavenet decoder

    Cristina Gârbacea, Aäron van den Oord, Yazhe Li, Felicia SC Lim, Alejandro Luebs, Oriol Vinyals, and Thomas C Walters. Low bit-rate speech coding with vq-vae and a wavenet decoder. InICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 735–739. IEEE,

    work page 2019

  14. [14]

    Audio set: An ontology and human-labeled dataset for audio events

    Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pp. 776–780. IEEE,

    work page 2017

  15. [15]

    It’s raw! audio generation with state-space models

    Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It’s raw! audio generation with state-space models. arXiv preprint arXiv:2202.09729 ,

    work page arXiv

  16. [16]

    Visqol: The virtual speech quality objective listener

    Andrew Hines, Jan Skoglund, Anil Kokaram, and Naomi Harte. Visqol: The virtual speech quality objective listener. In IW AENC 2012; International Workshop on Acoustic Signal Enhancement , pp. 1–4. VDE,

    work page 2012

  17. [17]

    Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. Technical Report 1502.03167, arXiv,

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    Architecture for variable bitrate neural speech codec with configurable computation complexity

    Tejas Jayashankar, Thilo Koehler, Kaustubh Kalgaonkar, Zhiping Xiu, Jilong Wu, Ju Lin, Prabhav Agrawal, and Qing He. Architecture for variable bitrate neural speech codec with configurable computation complexity. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 861–865. IEEE,

    work page 2022

  19. [19]

    End-to-end neural speech coding for real-time communications

    Xue Jiang, Xiulian Peng, Chengyu Zheng, Huaying Xue, Yuan Zhang, and Yan Lu. End-to-end neural speech coding for real-time communications. InICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 866–870. IEEE,

    work page 2022

  20. [20]

    arXiv preprint arXiv:2109.03264 , year=

    12 Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, et al. Text-free prosody-aware generative spoken language modeling. arXiv preprint arXiv:2109.03264 ,

    work page arXiv

  21. [21]

    Generative speech coding with predictive variance regularization

    W Bastiaan Kleijn, Andrew Storus, Michael Chinen, Tom Denton, Felicia SC Lim, Alejandro Luebs, Jan Skoglund, and Hengchin Yeh. Generative speech coding with predictive variance regularization. InICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 6478–6482. IEEE,

    work page 2021

  22. [22]

    Textless speech emotion conversion using decomposed and discrete representations.arXiv preprint arXiv:2111.07402 ,

    Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu-Anh Nguyen, Morgane Rivière, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, and Yossi Adi. Textless speech emotion conversion using decomposed and discrete representations.arXiv preprint arXiv:2111.07402 ,

    work page arXiv

  23. [23]

    Direct speech-to-speech translation with discrete units

    Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, et al. Direct speech-to-speech translation with discrete units. arXiv preprint arXiv:2107.05604, 2021a. Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Juan Pino, Jiatao Gu, and Wei-Ni...

    work page arXiv 2021

  24. [24]

    Robust low rate speech coding based on cloned networks and wavenet

    Felicia SC Lim, W Bastiaan Kleijn, Michael Chinen, and Jan Skoglund. Robust low rate speech coding based on cloned networks and wavenet. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 6769–6773. IEEE,

    work page 2020

  25. [25]

    Speech enhancement for low bit rate speech codec

    Ju Lin, Kaustubh Kalgaonkar, Qing He, and Xin Lei. Speech enhancement for low bit rate speech codec. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 7777–7781. IEEE,

    work page 2022

  26. [26]

    Generative spoken dialogue language modeling

    Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, et al. Generative spoken dialogue language modeling. arXiv preprint arXiv:2203.16502 ,

    work page arXiv

  27. [27]

    Disentangling speech from surroundings in a neural audio codec

    Ahmed Omran, Neil Zeghidour, Zalán Borsos, Félix de Chaumont Quitry, Malcolm Slaney, and Marco Tagliasacchi. Disentangling speech from surroundings in a neural audio codec. arXiv preprint arXiv:2203.15578,

    work page arXiv

  28. [28]

    WaveNet: A Generative Model for Raw Audio

    Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio.arXiv preprint arXiv:1609.03499,

    work page internal anchor Pith review Pith/arXiv arXiv

  29. [29]

    arXiv preprint arXiv:2104.00355 , year=

    Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrah- man Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprint arXiv:2104.00355 ,

    work page arXiv

  30. [30]

    Enhanced direct speech-to-speech translation using self-supervised pre-training and data augmentation

    Sravya Popuri, Peng-Jen Chen, Changhan Wang, Juan Pino, Yossi Adi, Jiatao Gu, Wei-Ning Hsu, and Ann Lee. Enhanced direct speech-to-speech translation using self-supervised pre-training and data augmentation. arXiv preprint arXiv:2204.02967 ,

    work page arXiv

  31. [31]

    Improving opus low bit rate quality with neural speech synthesis.arXiv preprint arXiv:1905.04628,

    Jan Skoglund and Jean-Marc Valin. Improving opus low bit rate quality with neural speech synthesis.arXiv preprint arXiv:1905.04628,

    work page arXiv 1905

  32. [32]

    Seanet: A multi- modal speech enhancement network,

    Marco Tagliasacchi, Yunpeng Li, Karolis Misiunas, and Dominik Roblek. Seanet: A multi-modal speech enhancement network. arXiv preprint arXiv:2009.02095 ,

    work page arXiv 2009

  33. [33]

    Lpcnet: Improving neural speech synthesis through linear prediction

    Jean-Marc Valin and Jan Skoglund. Lpcnet: Improving neural speech synthesis through linear prediction. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 5891–5895. IEEE, 2019a. Jean-Marc Valin and Jan Skoglund. A real-time wideband neural vocoder at 1.6 kb/s using lpcnet.arXiv preprint arXiv:1903.1...

    work page internal anchor Pith review arXiv 2019

  34. [34]

    Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram

    Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 6199–6203. IEEE, 2020a. Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Par...

    work page arXiv 2020

  35. [35]

    License with asterisk annotation * imply that the specific license varies across the dataset and is specific to each sample

    15 Table A.1: Datasets description. License with asterisk annotation * imply that the specific license varies across the dataset and is specific to each sample. Dataset Audio domain Sampling rate Channels Duration License Common Voice 7.0 Speech 48 kHz 1 9,096 h CC-0 DNS Challenge 4 (speech) Speech 48 kHz 1 2,425 h Multiples* AudioSet General audio 48 kHz 2...

    work page 2021

  36. [36]

    We used LeakyReLU as non-linear activation function

    was the only one that prevented the discriminator from diverging. We used LeakyReLU as non-linear activation function. Finally, training hyper parameters are not shared either so we use the same parameters as for ourEnCodec model. A.2 Alternative quantizers A.2.1 DiffQ Quantizer Pseudo quantization noise.We perform scalar quantization of the latent represe...

    work page 1996

  37. [37]

    We extend the DiffQ approach for latent space quantization, adding support for streamable rescaling, proper sparsity, and improved prior coding

    with a differentiable bandwidth estimate. We extend the DiffQ approach for latent space quantization, adding support for streamable rescaling, proper sparsity, and improved prior coding. Formally, we introduce a learnt parameterB∈ RD (with D the dimension of the latent space) such that B(i) represents the number of bits to use of thei-th dimension. In pract...

    work page 2015

  38. [38]

    This gives us a differentiable approximately 1-hot vector over the codebooks, i.e., notingGS the gumbel-softmax, zq,train = NC∑ i=1 GS(log(qi(z)),τ )T Ci

    with a temperatureτ = 0.5. This gives us a differentiable approximately 1-hot vector over the codebooks, i.e., notingGS the gumbel-softmax, zq,train = NC∑ i=1 GS(log(qi(z)),τ )T Ci. (8) At test time, we replace the gumbel-softmax with a sampling from the distributionqi. We define for alli, pi = softmax(li) the prior distribution over the codebooks entries w...

    work page 2021