pith. sign in

arxiv: 2310.09424 · v1 · pith:Z3ZJ6F7Lnew · submitted 2023-10-13 · 💻 cs.CL · cs.HC· cs.SD· eess.AS

SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translation

Zhehuai Chen , He Huang , Andrei Andrusenko , Oleksii Hrinchuk , Krishna C. Puvvada , Jason Li , Subhankar Ghosh , Jagadeesh Balam
show 1 more author
Boris Ginsburg
This is my paper
classification 💻 cs.CL cs.HCcs.SDeess.AS
keywords speechin-contextlearningsalmmodelcapabilitieslanguageproposed
0
0 comments X
read the original abstract

We present a novel Speech Augmented Language Model (SALM) with {\em multitask} and {\em in-context} learning capabilities. SALM comprises a frozen text LLM, a audio encoder, a modality adapter module, and LoRA layers to accommodate speech input and associated task instructions. The unified SALM not only achieves performance on par with task-specific Conformer baselines for Automatic Speech Recognition (ASR) and Speech Translation (AST), but also exhibits zero-shot in-context learning capabilities, demonstrated through keyword-boosting task for ASR and AST. Moreover, {\em speech supervised in-context training} is proposed to bridge the gap between LLM training and downstream speech tasks, which further boosts the in-context learning ability of speech-to-text models. Proposed model is open-sourced via NeMo toolkit.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Phonemes vs. Projectors: An Investigation of Speech-Language Interfaces for LLM-based ASR

    eess.AS 2026-04 unverdicted novelty 7.0

    Phoneme-based interfaces match or surpass projector-based ones for LLM ASR, especially in low-resource languages, and a BPE-phoneme hybrid offers additional improvements.

  2. TRADE: Transducer-Augmented Decoder for Speech LLM

    cs.CL 2026-06 unverdicted novelty 6.0

    TRADE augments multimodal Speech LLMs with a transducer branch for streaming ASR, reporting 6.71% WER offline and 8.40% streaming on the Open ASR Leaderboard from one checkpoint.

  3. Safety-Oriented Evaluation of Language Understanding Systems for Air Traffic Control

    cs.CL 2026-05 unverdicted novelty 6.0

    A consequence-aware evaluation framework applied to LLMs in ATC finds peak Risk Score of only 0.69 despite high macro-F1, with errors concentrated in high-impact entities.

  4. Afrispeech Semantics: Evaluating Audio Semantic Reasoning in Spoken Language Models Across Domains and Accents

    cs.CL 2026-05 unverdicted novelty 4.0

    Audio language models are benchmarked on five semantic and paralinguistic reasoning tasks to reveal limitations in handling spoken audio evidence, accent variation, and domain shifts.