pith. sign in

arxiv: 2605.28084 · v1 · pith:YLGDYJLBnew · submitted 2026-05-27 · 💻 cs.CL · cs.AI

SMILE-Next: Teaching Large Language Models to Detect, Classify, and Reason about Laughter

Lee Jung-Mok , Kim Sung-Bin , Joohyun Chang , Lee Hyun , Tae-Hyun Oh This is my paper

Pith reviewed 2026-06-29 13:23 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords laughter understandingmultimodal LLMself-instructmixture of expertslaughter detectionlaughter classificationlaughter reasoningreal-world scenarios
0
0 comments X

The pith

A new dataset plus automatic instruction synthesis and expert routing lets large language models outperform baselines on real-world laughter detection, classification, and reasoning.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper creates SMILE-Next, a dataset of multimodal data and question-answer pairs covering laughter detection, type classification, and reasoning about communicative intent in everyday contexts. It adds laughter-specific Self-Instruct to generate varied training instructions automatically and introduces the Mixture-of-Laugh-Experts framework that routes each task to a dedicated expert. Experiments show the combination produces stronger results than standard multimodal large language models. Readers would care because laughter functions as a social signal that carries meaning beyond amusement, so better machine handling of it could aid conversation analysis and media processing.

Core claim

SMILE-Next supplies multimodal textual representations and question-answer annotations for three tasks: laughter detection, laughter type classification, and laughter reasoning in real-world scenarios. Laughter-specific Self-Instruct automatically synthesizes diverse laughter-centric instructions to improve generalization across tasks and domains. The Mixture-of-Laugh-Experts framework adds a task-adaptive expert routing mechanism that dynamically selects specialized experts for each laughter-related task, and the combination of these components substantially outperforms multimodal LLM baselines.

What carries the argument

Mixture-of-Laugh-Experts (MoLE) framework, which uses task-adaptive expert routing to dynamically select specialized experts tailored to each laughter-related task.

If this is right

  • Models gain higher accuracy at detecting laughter presence in multimodal video and text inputs.
  • Classification of distinct laughter types becomes more reliable.
  • Reasoning about the underlying communicative intent of laughter improves.
  • Task efficiency rises because only the relevant expert activates for each input rather than running a single large model on every case.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The routing approach could transfer to other nonverbal social signals such as tone shifts or gesture sequences without requiring an entirely new architecture.
  • SMILE-Next could function as a shared benchmark for testing future multimodal models on social cue interpretation.
  • Whether the gains hold for live, unscripted conversations outside the dataset's distribution remains testable with independent recordings.

Load-bearing premise

The automatically synthesized laughter-centric instructions and the task-adaptive expert routing in MoLE produce genuine generalization rather than dataset-specific fitting.

What would settle it

Evaluating the resulting model on a fresh collection of laughter examples drawn from video sources or cultural settings absent from SMILE-Next and checking whether the reported performance gains over baselines disappear.

Figures

Figures reproduced from arXiv: 2605.28084 by Joohyun Chang, Kim Sung-Bin, Lee Hyun, Lee Jung-Mok, Tae-Hyun Oh.

Figure 1
Figure 1. Figure 1: Task overview of SMILE-Next. We present SMILE-Next, a comprehensive dataset for laughter understanding. The dataset includes three fundamental tasks for understanding laughter in social interactions: laughter detection, laughter type classification, and laughter reasoning. ter expert Large Language Model (LLM), de￾signed to identify and interpret laughter in social interaction. Because laughter arises from… view at source ↗
Figure 2
Figure 2. Figure 2: Architecture of Mixture-of-Laugh-Experts. LoRA-based expert modules are added to a frozen pre￾trained weights and are dynamically weighted by a router gating network, resulting in laughter task-specific specialization. This framework leverages an external LLM (We use GPT-4 API) to generate both instructions and responses. This approach facilitates a scalable data acquisition pipeline, enabling rich knowled… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative results. As shown, LLMs trained at SMILE-Next can reason and analyze about the ambiguous and slight laughter, exactly giving the reason and classifying what this laugh is. Comparing AV-LLM and V-LLM to the LLMs which uses textual multimodal cues, it can explicitly point out to analyze why this laugh has occurred. factor α set to 16 for each expert. The models are supervised fine-tuned with the … view at source ↗
Figure 4
Figure 4. Figure 4: Visualization of MoLE’s router weights. The results show dominant activation of Expert 1, with task-dependent variations, classification assigns more weight to Expert 2, while detection does so for Expert 3. Detect. (ms) Cls. (ms) Reasoning (ms) All (ms) Single expert 981 790 2802 1494 Multi-experts(MoLE) 991 796 2845 1513 Difference +10 +6 +43 +19 [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: SMILE-Next statistics. We report the num￾ber of utterances and the video duration of each video samples in SMILE-Next at top row, which shows similar distribution. At bottom row, we also report the length of input and output text, and as shown input prompt is diverse while the output text shows distribution centered at 200. clip is different, because the type of conversa￾tion from the source is different. … view at source ↗
Figure 6
Figure 6. Figure 6 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Additional qualitative results and example of textualized multimodal cues [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Example screenshot of Amazon Mechanical Turk for human annotation. For laughter classification, the human evaluators are instructed to pick a laughter type with a confident, and to write the reason why they choose it [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
read the original abstract

Laughter is a complex social signal that conveys communicative intent beyond amusement. While prior work has focused on isolated laughter analysis tasks, a comprehensive understanding of laughter in real-world scenarios remains underexplored. Therefore, we introduce SMILE-Next, a dataset for real-world laughter understanding with multimodal textual representations and question-answer annotations across three tasks: laughter detection, laughter type classification, and laughter reasoning. Building upon SMILE-Next, we aim to develop a laughter-specialized large language model capable of nuanced understanding of laughter in real-world contexts. To this end, we propose two key components: laughter-specific Self-Instruct and the Mixture-of-Laugh-Experts (MoLE) framework. Laughter-specific Self-Instruct enhances generalization across tasks and domains by automatically synthesizing diverse laughter-centric instructions. MoLE introduces a task-adaptive expert routing mechanism that dynamically selects specialized experts tailored to each laughter-related task, improving task-specific performance and efficiency. Experimental results show that the combination of our proposed components substantially outperforms multimodal LLM baselines, advancing robust real-world laughter understanding. Project page is at: https://mok0102.github.io/smile-next/.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces the SMILE-Next dataset, which provides multimodal textual representations and QA annotations for three laughter-related tasks (detection, type classification, and reasoning). It proposes laughter-specific Self-Instruct to automatically synthesize diverse instructions and the Mixture-of-Laugh-Experts (MoLE) framework with task-adaptive expert routing. The central claim is that the combination of these components substantially outperforms multimodal LLM baselines on the proposed tasks.

Significance. If the reported gains are shown to reflect genuine generalization rather than fitting to the automatically synthesized data, the work would provide a useful specialized dataset and routing architecture for social-signal understanding, with potential relevance to affective computing and multimodal dialogue systems.

major comments (2)
  1. [Abstract] Abstract: the claim that the proposed components 'substantially outperforms multimodal LLM baselines' is stated without any quantitative metrics, baseline names, error bars, or statistical tests, which is load-bearing for the central claim and prevents assessment of whether the result holds.
  2. [Experimental Results] The evaluation is performed only on the newly constructed SMILE-Next corpus; without reported ablations that isolate the Self-Instruct synthesis step, comparisons against human-authored instructions, or results on an external laughter corpus, the outperformance result remains compatible with dataset-specific fitting rather than robust generalization from the MoLE routing mechanism.
minor comments (2)
  1. [Abstract] The abstract mentions 'multimodal textual representations' but does not clarify how audio or visual laughter cues are converted into text or whether this conversion introduces information loss.
  2. [Method] Notation for the MoLE routing mechanism should be defined more explicitly (e.g., how task embeddings are computed and how experts are selected) to allow replication.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the constructive feedback on the abstract and evaluation design. We address each major comment below and will revise the manuscript to improve clarity and address concerns about the strength of the claims.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claim that the proposed components 'substantially outperforms multimodal LLM baselines' is stated without any quantitative metrics, baseline names, error bars, or statistical tests, which is load-bearing for the central claim and prevents assessment of whether the result holds.

    Authors: We agree that the abstract should provide quantitative support for the central claim. In the revised version, we will update the abstract to include specific performance metrics, name the multimodal LLM baselines compared, and reference error bars or statistical tests from the experimental results section. revision: yes

  2. Referee: [Experimental Results] The evaluation is performed only on the newly constructed SMILE-Next corpus; without reported ablations that isolate the Self-Instruct synthesis step, comparisons against human-authored instructions, or results on an external laughter corpus, the outperformance result remains compatible with dataset-specific fitting rather than robust generalization from the MoLE routing mechanism.

    Authors: The manuscript includes ablations demonstrating the contribution of the MoLE routing mechanism. We agree that further isolating the Self-Instruct synthesis step and adding comparisons against human-authored instructions would strengthen the evidence for generalization. We will add these ablations in the revision. Results on an external laughter corpus are not currently available and would require new data collection. revision: partial

standing simulated objections not resolved
  • Results on an external laughter corpus (would require new data collection beyond the scope of the current work)

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper introduces a new dataset (SMILE-Next) with synthesized QA annotations and proposes two components (laughter-specific Self-Instruct for instruction synthesis and MoLE for task-adaptive routing). No equations, parameter-fitting procedures, or self-citation chains appear in the abstract or described methods that would reduce any claimed prediction or result to the inputs by construction. The central claim of outperformance is framed as an empirical comparison against multimodal LLM baselines, which remains externally falsifiable and does not rely on self-referential definitions or renamings. The derivation chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only view supplies no explicit free parameters, axioms, or invented entities; ledger left empty.

pith-pipeline@v0.9.1-grok · 5746 in / 916 out tokens · 27835 ms · 2026-06-29T13:23:27.328414+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

8 extracted references · 4 canonical work pages · 2 internal anchors

  1. [1]

    arXiv preprint arXiv:2312.09979 , year=

    Modeling prosodic features with joint factor analysis for speaker verification.IEEE Transactions on Audio, Speech, and Language Processing. Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, and 1 others. 2023. Loramoe: Alleviate world knowledge forgetting in large lan- guage models via moe-st...

    work page arXiv 2023

  2. [2]

    VisualBERT: A Simple and Performant Baseline for Vision and Language

    TVQA: Localized, compositional video ques- tion answering. InProceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 1369–1379, Brussels, Belgium. Association for Computational Linguistics. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng ...

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  3. [3]

    arXiv preprint arXiv:2407.12927 , year=

    Vilbert: Pretraining task-agnostic visiolinguis- tic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems (NeurIPS). Anton Nijholt, Andreea I Niculescu, Alessandro Vali- tutti, and Rafael E Banchs. 2017. Humor in human- computer interaction: a short survey.Adjunct Pro- ceedings of INTERACT. Kishore Papineni, ...

    work page arXiv 2017

  4. [4]

    Caleb Warren, Adam Barsky, and A Peter McGraw

    An explanation of the relationship between instructor humor and student learning: Instructional humor processing theory.Communication education. Caleb Warren, Adam Barsky, and A Peter McGraw

  5. [5]

    Personality and Social Psychology Review

    What makes things funny? an integrative review of the antecedents of laughter and amusement. Personality and Social Psychology Review. Orion Weller and Kevin Seppi. 2020. The rJokes dataset: a large scale humor collection. InProceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 6136–6141, Marseille, France. European Language Res...

    2020

  6. [6]

    Qwen2.5-Omni Technical Report

    Supporting artificial social intelligence with theory of mind.Frontiers in artificial intelligence. Binzhu Xie, Sicheng Zhang, Zitang Zhou, Bo Li, Yuan- han Zhang, Jack Hessel, Jingkang Yang, and Ziwei Liu. 2024. Funqa: Towards surprising video com- prehension. InEuropean Conference on Computer Vision, pages 39–57. Springer. Jin Xu, Zhifang Guo, Jinzheng ...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  7. [7]

    For each segments, we also leveraged Active Speaker Algorithm (Tao et al., 2021) using audio-visual information in each video

    and trimmed a video into multiple segments; one utterance for one segment. For each segments, we also leveraged Active Speaker Algorithm (Tao et al., 2021) using audio-visual information in each video. For the visual cue, we acquired video cap- tion for each segment, which explains each seg- ment’s situation, providing the visual cue. We also extracted th...

    2021

  8. [8]

    high pitch

    of speakers in the video; Accumulating each frame’s facial action and selected most dominant top-3 facial action units for each person. We ex- tract acoustic features from speech utterances using off-the-shelf speech processing tools (Dehak et al., 2007), including the mean and variance of pitch and intensity, as well as jitter and shimmer. Al- though we ...

    2007