pith. sign in

arxiv: 2606.26342 · v1 · pith:YFTYB2HBnew · submitted 2026-06-24 · 📡 eess.AS

A Large-Scale Database and Predictive Model of Listener-Rated Ease of Speech Understanding in Commercial Hearing Aids

Andrew Sabin , Steve Taddei , Abram Bailey This is my paper

Pith reviewed 2026-06-26 01:08 UTC · model grok-4.3

classification 📡 eess.AS
keywords hearing aidslistener ratingsease of understandingWhisper encoderperceptual metriccommercial devicesHASPIv2speech understanding
0
0 comments X

The pith

A model using differences in Whisper encoder representations predicts listener-rated ease of understanding for commercial hearing aids better than HASPIv2.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper collects 151,608 ratings from website visitors with self-reported hearing loss on binaural recordings of 83 commercial hearing aids across 72 realistic scenes, yielding over 100,000 screened ratings on a five-point ease-of-understanding scale. It then trains a small MLP on the difference between Whisper encoder embeddings of the aided audio and a clean reference to predict those ratings. On held-out devices the resulting metric reaches correlations of 0.92 overall with listener ratings, compared with 0.83 for HASPIv2, and matches split-half listener reliability in loud scenes. A sympathetic reader would care because the work supplies both a public-scale perceptual dataset and a practical predictor tied directly to how real users experience commercial products rather than to simulated intelligibility.

Core claim

The central claim is that a learned metric formed by subtracting frozen Whisper encoder representations of aided commercial hearing-aid audio from a matched clean-speech reference, then passing the difference through a small MLP, predicts listener-rated ease of understanding at the scene level with higher correlation than HASPIv2 (overall 0.92 versus 0.83) on held-out devices, reaching the split-half reliability of the ratings themselves in loud scenes and approaching it in quiet scenes while also responding appropriately to controlled gain and SNR changes.

What carries the argument

The central mechanism is the difference embedding obtained by subtracting internal representations of the aided recording from those of the clean reference inside a frozen Whisper encoder, followed by a small MLP head trained to map that embedding to the five-point listener ease ratings.

If this is right

  • The metric can evaluate new commercial hearing-aid devices on existing recordings without additional listener testing.
  • It reaches the reliability ceiling of averaged human ratings in loud scenes.
  • The model changes sensibly when gain or SNR is altered in controlled tests.
  • The dataset supplies a new benchmark for comparing real commercial products on perceived speech ease.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same difference-embedding approach could be tested on other classes of audio devices that alter speech, such as cochlear implants or public-address systems.
  • If the full set of screened recordings and ratings is made available, researchers could train alternative heads or compare additional embedding models directly against the same human data.
  • The gap between loud-scene and quiet-scene performance suggests that background noise may be the dominant factor the Whisper difference captures, which could be verified by ablating noise-only scenes.

Load-bearing premise

The online MUSHRA-style blind tests by self-reported hearing-loss participants produce ratings that are stable, unbiased, and representative of real-world ease of understanding for the 83 commercial products tested.

What would settle it

New listener ratings collected on a fresh set of commercial hearing-aid recordings from devices and scenes absent from the training data show the learned metric correlating no higher than 0.83 with those ratings.

Figures

Figures reproduced from arXiv: 2606.26342 by Abram Bailey, Andrew Sabin, Steve Taddei.

Figure 1
Figure 1. Figure 1: Model overview. Processed (aided) audio and a matched clean reference pass through the same frozen 12-layer Whisper-small encoder; the selected layer’s hidden states are mean-pooled over time and differenced to form a 768-D difference embedding, which the trained MLP head maps to a predicted rating. Scene level selects the encoder layer (blue: layer 2, quiet; red: layer 5, loud). 4 [PITH_FULL_IMAGE:figure… view at source ↗
Figure 2
Figure 2. Figure 2: Each metric versus the mean listener rating of ease of speech understanding at the talker-pooled scene level, colored by scene type. Left: HASPIv2 correlates moderately in loud scenes and weakly in quiet scenes. Right: the learned metric, on devices held out of training, tracks the human ratings closely. 0 3 6 9 > 1 kHz target undershoot (dB) 0.2 0.0 0.2 0.4 predicted rating Gain Loud Quiet 0 3 6 9 SNR boo… view at source ↗
Figure 3
Figure 3. Figure 3: Change in predicted ease of speech understanding as synthetic devices undershoot the NAL-NL2 gain target (left) or receive an SNR boost via noise attenuation (right). Gain undershoot hurts both scene types; added SNR helps mainly in loud scenes. Bands show the spread across scenes. production-trained (all-data) model while varying two interpretable dimensions: (1) fit-to-target gain, by in￾tentionally unde… view at source ↗
read the original abstract

HearAdvisor aims to provide hearing-aid consumers with audio-performance metrics and recordings that reflect real listening experience. For speech-related metrics, HearAdvisor has historically used HASPIv2, a metric designed to predict objective intelligibility and validated primarily under simulated distortions. Its relationship to consumer-rated ease of understanding for commercial hearing aids is uncertain. Here we introduce a large-scale perceptual dataset and learned metric for listener-rated perceived benefit for speech understanding. Website visitors with self-reported hearing loss completed a blind, MUSHRA-inspired listening test in which they rated recordings of commercial hearing aids on a five-point "Ease of Understanding" scale. The dataset contains 151,608 ratings, 104,298 after quality screening, spanning 10,394 binaural acoustic-manikin recordings from 83 commercial products across 72 realistic acoustic scenes. To predict these ratings, we pass aided audio and a matched clean-speech reference through a frozen Whisper encoder, subtract their internal representations, and train a small MLP head on the resulting difference embedding. On devices held out of training, the learned metric substantially outperforms HASPIv2 at the scene level (overall r = 0.92 vs. 0.83; loud = 0.89 vs. 0.75; quiet = 0.79 vs. 0.58). In loud scenes, performance reaches the split-half reliability of the listener ratings; in quiet scenes, it approaches that ceiling. The model also responds sensibly to controlled gain and SNR manipulations. Together, the dataset and model provide a new way to predict listener-rated ease of speech understanding for real commercial hearing-aid recordings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The paper introduces a dataset of 151,608 (104,298 post-screening) listener ratings of ease-of-understanding for 10,394 binaural recordings from 83 commercial hearing aids across 72 scenes, collected via an online MUSHRA-inspired 5-point scale from self-reported hearing-loss participants. It trains a small MLP head on the difference between frozen Whisper encoder embeddings of aided and clean reference audio to predict these ratings. On devices held out of training, the model reports scene-level Pearson correlations of 0.92 (overall), 0.89 (loud), and 0.79 (quiet) versus HASPIv2 values of 0.83/0.75/0.58, reaching split-half listener reliability in loud scenes and responding appropriately to gain/SNR changes.

Significance. If the evaluation holds, the work supplies a perceptually grounded, data-driven alternative to HASPIv2 for commercial hearing-aid assessment at large scale; the held-out-device protocol and controlled-manipulation checks supply partial independent grounding beyond direct fitting to ratings. The dataset size itself is a notable resource for the field.

major comments (3)
  1. [Abstract] Abstract: the headline correlations (r = 0.92 vs. 0.83 overall; 0.89 vs. 0.75 loud) are presented without training-procedure details, data-partitioning protocol, statistical significance tests, or error bars, rendering the numerical support for the central outperformance claim unevaluable.
  2. [Abstract] Abstract: the claim that performance 'reaches the split-half reliability of the listener ratings' in loud scenes supplies no description of how split-half reliability was computed or how the comparison was performed, which is load-bearing for the ceiling-performance assertion.
  3. [Abstract] Abstract: the entire evaluation rests on ratings obtained from an uncontrolled web-based protocol with self-reported (unverified) hearing loss and personal devices/environments; no audiometric validation, test-retest reliability, or bias analysis is reported, directly affecting whether the learned metric can be said to predict real listener-rated benefit.
minor comments (1)
  1. The manuscript should clarify the exact Whisper model variant, layer(s) used for the difference embedding, and any preprocessing of the 10,394 recordings.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback on the abstract. The full manuscript provides the requested methodological and statistical details in Sections 3 and 4, but we address each point below and indicate where revisions can strengthen clarity.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the headline correlations (r = 0.92 vs. 0.83 overall; 0.89 vs. 0.75 loud) are presented without training-procedure details, data-partitioning protocol, statistical significance tests, or error bars, rendering the numerical support for the central outperformance claim unevaluable.

    Authors: The abstract is intentionally concise. The training procedure (difference of frozen Whisper embeddings fed to an MLP), held-out device partitioning protocol, and statistical comparisons (including significance and error bars) are fully detailed in Sections 3.2, 4.1, and the results figures/tables. We will revise the abstract to include a brief qualifier on the held-out-device evaluation protocol. revision: partial

  2. Referee: [Abstract] Abstract: the claim that performance 'reaches the split-half reliability of the listener ratings' in loud scenes supplies no description of how split-half reliability was computed or how the comparison was performed, which is load-bearing for the ceiling-performance assertion.

    Authors: Split-half reliability is computed in Section 4.3 by randomly partitioning listener ratings per scene into two halves, correlating the halves, and averaging over multiple splits; the model is then compared directly to this value. We will add a short clarifying clause to the abstract if length permits. revision: partial

  3. Referee: [Abstract] Abstract: the entire evaluation rests on ratings obtained from an uncontrolled web-based protocol with self-reported (unverified) hearing loss and personal devices/environments; no audiometric validation, test-retest reliability, or bias analysis is reported, directly affecting whether the learned metric can be said to predict real listener-rated benefit.

    Authors: The online protocol with self-reported hearing loss is a deliberate design choice to capture real consumer experiences with personal devices, as discussed in the Limitations section. Quality screening was applied and controlled gain/SNR checks were performed. We will expand the abstract and discussion to explicitly note the absence of audiometric verification and to reference the bias analysis already present in the supplementary material. revision: yes

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper trains a Whisper-difference MLP explicitly on listener ratings from a subset of devices, then reports correlations on held-out devices (r=0.92 overall) and compares to the independent HASPIv2 metric. This is standard supervised generalization testing rather than a reduction by construction. No equations, self-citations, uniqueness theorems, or ansatzes are invoked that collapse the central claim to its inputs. Additional checks (gain/SNR response, split-half reliability) provide external grounding. The derivation is self-contained against the stated benchmarks.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim rests on the validity of the collected listener ratings as ground truth and on the assumption that Whisper embeddings contain the relevant perceptual differences; the MLP weights are the only free parameters introduced by the work itself.

free parameters (1)
  • MLP head weights and biases
    Small neural network trained to map Whisper difference embeddings to five-point ease ratings.
axioms (1)
  • domain assumption Frozen Whisper encoder representations capture speech features relevant to listener-rated ease of understanding
    The method subtracts Whisper internal representations without any fine-tuning or justification that these features align with subjective ease.

pith-pipeline@v0.9.1-grok · 5833 in / 1435 out tokens · 31077 ms · 2026-06-26T01:08:06.689296+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 1 canonical work pages

  1. [1]

    Barker, M

    J. Barker, M. A. Akeroyd, T. J. Cox, J. F. Culling, J. Firth, S. Graetzer, H. Griffiths, L. Harris, G. Naylor, Z. Podwinska, E. Porter, and R. Viveros Mu\ noz. The 1st Clarity Prediction Challenge: A machine learning challenge for hearing aid intelligibility prediction. Proc.\ Interspeech, pages 3508--3512, 2022

    2022

  2. [2]

    Bisgaard, M

    N. Bisgaard, M. S. Vlaming, and M. Dahlquist. Standard audiograms for the IEC 60118-15 measurement procedure. Trends in Amplification, 14(2):113--120, 2010

    2010

  3. [3]

    Recommendation BS.1534-3: Method for the subjective assessment of intermediate quality level of audio systems

    ITU-R. Recommendation BS.1534-3: Method for the subjective assessment of intermediate quality level of audio systems. International Telecommunication Union, Geneva, 2015

    2015

  4. [4]

    J. M. Kates and K. H. Arehart. The Hearing-Aid Speech Perception Index (HASPI) version 2. Speech Communication, 131:35--46, 2021

    2021

  5. [5]

    Keidser, H

    G. Keidser, H. Dillon, M. Flax, T. Ching, and S. Brewer. The NAL-NL2 prescription procedure. Audiology Research, 1(1):e24, 88--90, 2011

    2011

  6. [6]

    \'E. Lombard. Le signe de l'\'el\'evation de la voix. Annales des Maladies de l'Oreille et du Larynx, 37:101--119, 1911

    1911

  7. [7]

    Manchaiah, S

    V. Manchaiah, S. Taddei, A. Bailey, D. W. Swanepoel, H. Rodrigo, and A. Sabin. A novel consumer-centric metric for evaluating hearing device audio performance. Frontiers in Audiology and Otology, 2:1406362, 2024

    2024

  8. [8]

    Radford, J

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. In Proc.\ ICML, volume 202 of PMLR, pages 28492--28518, 2023

    2023

  9. [9]

    Sabin, S

    A. Sabin, S. Taddei, and A. Bailey. HearAdvisor procedures for recording and evaluating hearing devices v1.0. White paper, OSF Preprints, 2023. https://doi.org/10.31219/osf.io/trnga

    work page doi:10.31219/osf.io/trnga 2023

  10. [10]

    Weisser, J

    A. Weisser, J. M. Buchholz, C. Oreinos, J. Williges, J. Galloway, T. Humburg, T. Galloway, and H. Dillon. The Ambisonic Recordings of Typical Environments (ARTE) database. Acta Acustica united with Acustica, 105(4):695--713, 2019

    2019

  11. [11]

    Y.-H. Wu, E. Stangl, O. Chipara, S. S. Hasan, A. Welhaven, and J. Oleson. Characteristics of real-world signal-to-noise ratios and speech listening situations of older adults with mild-to-moderate hearing loss. Ear and Hearing, 39(2):293--304, 2018

    2018

  12. [12]

    H. Yu, H. Zhou, L. Li, B. Cao, C. Mo, and S. X. Wang. Intrusive intelligibility prediction with ASR encoders. In Proc.\ 3rd Clarity Prediction Challenge (CPC3), 2025

    2025