CoChat

Identifying bias in models that detect vocal fold paralysis from audio recordings using explainable machine learning and clinician ratings

Open access

The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.

Abstract

Detecting voice disorders from voice recordings could allow for frequent, remote, and low-cost screening before costly clinical visits and a more invasive laryngoscopy examination. Our goals were to detect unilateral vocal fold paralysis (UVFP) from voice recordings using machine learning, to identify which acoustic variables were important for prediction to increase trust, and to determine model performance relative to clinician performance. Patients with confirmed UVFP through endoscopic examination (N = 77) and controls with normal voices matched for age and sex (N = 77) were included. Voice samples were elicited by reading the Rainbow Passage and sustaining phonation of the vowel "a". Four machine learning models of differing complexity were used. SHapley Additive exPlanations (SHAP) was used to identify important features. The highest median bootstrapped ROC AUC score was 0.87 and beat clinician’s performance (range: 0.74–0.81) based on the recordings. Recording durations were different between UVFP recordings and controls due to how that data was originally processed when storing, which we can show can classify both groups. And counterintuitively, many UVFP recordings had higher intensity than controls, when UVFP patients tend to have weaker voices, revealing a dataset-specific bias which we mitigate in an additional analysis. We demonstrate that recording biases in audio duration and intensity created dataset-specific differences between patients and controls, which models used to improve classification. Furthermore, clinician’s ratings provide further evidence that patients were over-projecting their voices and being recorded at a higher amplitude signal than controls. Interestingly, after matching audio duration and removing variables associated with intensity in order to mitigate the biases, the models were able to achieve a similar high performance. We provide a set of recommendations to avoid bias when building and evaluating machine learning models for screening in laryngology.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pdig.0000516

Cites

70 of 70 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig 1 Schematic of speech production and the process of extracting certain acoustic features from an audio signal. (A) Speech production, (B) recording characteristics, (C) waveform of audio signal with fundamental frequency (f0), (D) spectrogram with formants F1-F3 and intensity, (E) mel-frequency cepstral coefficients (MFCCs). Full description in the main text.
Fig 1 Schematic of speech production and the process of extracting certain acoustic features from an audio signal. (A) Speech production, (B) recording characteristics, (C) waveform of audio signal with fundamental frequency (f0), (D) spectrogram with formants F1-F3 and intensity, (E) mel-frequency cepstral coefficients (MFCCs). Full description in the main text.CC BY · PubMed Central
Fig 2 Distribution of audio duration for reading and vowel tasks split by group reveals a dataset bias. The mode of the audio durations for the controls is 3.5 s for reading samples and 4.11 s for vowel samples.
Fig 2 Distribution of audio duration for reading and vowel tasks split by group reveals a dataset bias. The mode of the audio durations for the controls is 3.5 s for reading samples and 4.11 s for vowel samples.CC BY · PubMed Central
Fig 3 Model performance comparison using a permutation test using non-redundant features. Scores from models trained on true labels (blue) and trained on permuted labels (orange) over bootstrapping splits.
Fig 3 Model performance comparison using a permutation test using non-redundant features. Scores from models trained on true labels (blue) and trained on permuted labels (orange) over bootstrapping splits.CC BY · PubMed Central
Fig 4 Feature importance parallel coordinate plot. Rank reads from bottom (most important) to top (least important). Mean rank is weighted by performance of each model to avoid a lower performing model biasing the mean rank.
Fig 4 Feature importance parallel coordinate plot. Rank reads from bottom (most important) to top (least important). Mean rank is weighted by performance of each model to avoid a lower performing model biasing the mean rank.CC BY · PubMed Central
Fig 5 Distributions for top 5 features and corresponding performance for single features. Logistic Regression with L1 penalty was used. No single feature is enough to dissociate groups with high performance. Null models’ median performance was 0.5.
Fig 5 Distributions for top 5 features and corresponding performance for single features. Logistic Regression with L1 penalty was used. No single feature is enough to dissociate groups with high performance. Null models’ median performance was 0.5.CC BY · PubMed Central
Fig 6 Feature redundancy with top 5 features highlighted. Top 5 features are highlighted in bold and their rank is displayed. Squares are clusters of redundant features. Computed with all participants on the reading task.
Fig 6 Feature redundancy with top 5 features highlighted. Top 5 features are highlighted in bold and their rank is displayed. Squares are clusters of redundant features. Computed with all participants on the reading task.CC BY · PubMed Central
Fig 7 Descriptive statistics and inter-rater reliability of clinician ratings for unilateral vocal fold paralysis (UVFP), background noise, and recording loudness indicating likely bias. Controls and UVFP are ground truth diagnosis from the full clinical interview. Ratings are on brief reading samples. Bars indicate maximum and minimum count across the three raters. The disproportionate amount of UVFP samples rated as having high background noise and high loudness indicates likely bias, where the gain might have been raised for some UVFP patients and they may have phonated more intensely. kappa: Light’s kappa; ICC: intra-class correlation coefficient.
Fig 7 Descriptive statistics and inter-rater reliability of clinician ratings for unilateral vocal fold paralysis (UVFP), background noise, and recording loudness indicating likely bias. Controls and UVFP are ground truth diagnosis from the full clinical interview. Ratings are on brief reading samples. Bars indicate maximum and minimum count across the three raters. The disproportionate amount of UVFP samples rated as having high background noise and high loudness indicates likely bias, where the gain might have been raised for some UVFP patients and they may have phonated more intensely. kappa: Light’s kappa; ICC: intra-class correlation coefficient.CC BY · PubMed Central
Fig 8 How clinicians rate the audio recordings of read speech: descriptive statistics and inter-rater reliability of average clinician ratings. The average across raters was taken for each recording. ICC: intra-class correlation coefficient.
Fig 8 How clinicians rate the audio recordings of read speech: descriptive statistics and inter-rater reliability of average clinician ratings. The average across raters was taken for each recording. ICC: intra-class correlation coefficient.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 5

  1. C. Molnar, 2019, Interpretable Machine Learning. Lulu.com
  2. N Sritharan, 2015, The vagus nerve
  3. S Ghosh Satrajit, Pydra-ML
  4. ZC Lipton, 2018, The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery., Queueing Syst., 16, 31
  5. M Gamer, 2012, Package “irr.”, Various coefficients of interrater reliability and agreement., 22, 1

Cited by

See all 16

Metadata

Authors
Low, D. M., Rao, V., Randolph, G., Song, P. C., & Ghosh, S. S.
Affiliations
  • Harvard University (US)
  • Massachusetts Eye and Ear Infirmary (US)
Corresponding author
Low, Daniel M.
Venue
PLOS Digital Health
Published
2024-05-30
Volume
3
Issue
5
Pages
e0000516
ISSN
2767-3170 (ISSN-L)
OpenAlex
W4399169406
Cited by
16
References
65
Citations by year
2026: 6 · 2025: 6 · 2024: 4
FWCI
4.55
Type
journal article
Language
en
Open access
Locations
  • PLOS Digital Health (publishedVersion, cc-by)
  • medRxiv (acceptedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion, cc-by-nc-nd)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
  • National Institute on Deafness and Other Communication Disorders (5T32DC000038-28)
  • RallyPoint Fellowship
  • Amelia Peabody Charitable Fund (Professional Development Award)
  • Gift to McGovern Institute for Brain Research at MIT
  • National Institute of Biomedical Imaging and Bioengineering (R01 EB020740, P41 EB019936)
  • NIH Office of the Director (1OT2OD032720-01)
Topics
  • Health Sciences › Health Professions › Speech and Hearing › Voice and Speech Disorders
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Health Sciences › Medicine › Pulmonary and Respiratory Medicine › Phonocardiography and Auscultation Techniques
Keywords
unilateral vocal fold paralysis, voice disorders, voice screening, acoustic voice analysis, explainable machine learning, SHAP, clinician ratings, data bias, recording duration, vocal intensity, laryngology, remote screening

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.