Identifying bias in models that detect vocal fold paralysis from audio recordings using explainable machine learning and clinician ratings
Open accessThe DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
Abstract
Detecting voice disorders from voice recordings could allow for frequent, remote, and low-cost screening before costly clinical visits and a more invasive laryngoscopy examination. Our goals were to detect unilateral vocal fold paralysis (UVFP) from voice recordings using machine learning, to identify which acoustic variables were important for prediction to increase trust, and to determine model performance relative to clinician performance. Patients with confirmed UVFP through endoscopic examination (N = 77) and controls with normal voices matched for age and sex (N = 77) were included. Voice samples were elicited by reading the Rainbow Passage and sustaining phonation of the vowel "a". Four machine learning models of differing complexity were used. SHapley Additive exPlanations (SHAP) was used to identify important features. The highest median bootstrapped ROC AUC score was 0.87 and beat clinician’s performance (range: 0.74–0.81) based on the recordings. Recording durations were different between UVFP recordings and controls due to how that data was originally processed when storing, which we can show can classify both groups. And counterintuitively, many UVFP recordings had higher intensity than controls, when UVFP patients tend to have weaker voices, revealing a dataset-specific bias which we mitigate in an additional analysis. We demonstrate that recording biases in audio duration and intensity created dataset-specific differences between patients and controls, which models used to improve classification. Furthermore, clinician’s ratings provide further evidence that patients were over-projecting their voices and being recorded at a higher amplitude signal than controls. Interestingly, after matching audio duration and removing variables associated with intensity in order to mitigate the biases, the models were able to achieve a similar high performance. We provide a set of recommendations to avoid bias when building and evaluating machine learning models for screening in laryngology.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures








Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“approximately 82% of the reviewed studies rely exclusively on sustained vowels, reporting binary detection accuracies that are, on average, 5 to 10 percentage points higher than those obtained via connected speech tasks in similar architectures [8,48].”
From Diagnostic Accuracy of Artificial Intelligence in Laryngeal Disorders: An Integrative Review (2026) · cites this paper as [8]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unilateral paralysis generates persistent glottic insufficiency and air leakage [4,8]”
From Diagnostic Accuracy of Artificial Intelligence in Laryngeal Disorders: An Integrative Review (2026) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirical evidence suggests that connected speech tasks provide superior diagnostic sensitivity, particularly for neuromuscular disorders and functional dysphonia, as they involve complex coordination and prosodic variations that are often masked during static phonation [8,9,19].”
From Diagnostic Accuracy of Artificial Intelligence in Laryngeal Disorders: An Integrative Review (2026) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unilateral vocal folds paralysis (UVFP) and spasmodic dysphonia also overlap acoustically, both yielding decreased harmonicity alongside increased glottic and subglottic noise [8].”
From Diagnostic Accuracy of Artificial Intelligence in Laryngeal Disorders: An Integrative Review (2026) · cites this paper as [8]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“…simpler and more complex models is important because simpler models such as Logistic Regression could be preferred because they tend to generalize better given they are less at risk for overfitting the training set and they are more interpretable and thus biases can be assessed more directly [68].”
Cicchetti, D. V. (1994) · Psychological Assessment · doi:10.1037/1040-3590.6.4.284 · Cited by 9,349
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial
Hallgren, K. A. (2012) · Tutorials in Quantitative Methods for Psychology · doi:10.20982/tqmp.08.1.p023 · Cited by 3,884
Measuring and testing dependence by correlation of distances
Székely, G. J., Rizzo, M. L., & Bakirov, N. K. (2007) · The Annals of Statistics · doi:10.1214/009053607000000505 · Cited by 2,684
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998
The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing
Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andre, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2457417 · Cited by 1,818 · cited 1×
“The audio samples were processed using OpenSmile with the eGeMAPS configuration file (article [34], source code [35]) which applies different summarization statistics to the time series depending on the feature resulting in 88 features per sample covering information related to the vocal folds (F0,…”
Thompson, C. G., Kim, R. S., Aloe, A. M., & Becker, B. J. (2017) · Basic and Applied Social Psychology · doi:10.1080/01973533.2016.1277529 · Cited by 1,092
Voice and Articulation Drillbook
Seashore, C. E. (1940) · Music Educators Journal · doi:10.2307/3388520 · Cited by 892
Research works printed but not linked: 5
- C. Molnar, 2019, Interpretable Machine Learning. Lulu.com
- N Sritharan, 2015, The vagus nerve
- S Ghosh Satrajit, Pydra-ML
- ZC Lipton, 2018, The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery., Queueing Syst., 16, 31
- M Gamer, 2012, Package “irr.”, Various coefficients of interrater reliability and agreement., 22, 1
Cited by
AI-readiness Criteria for Biomedical Data
Clark, T., Caufield, H., Parker, J. A., Al Manir, S., Amorim, E., Eddy, J., Gim, N., Gow, B., Goar, W., Hansen, J. N., et al. (2024) · doi:10.1101/2024.10.23.619844 · Cited by 32
Self-citationSpeech-Based Depression Assessment: A Comprehensive Survey
Leal, S. S., Ntalampiras, S., & Sassi, R. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3521327 · Cited by 41
Kim, Y. E., Dobko, M., Li, H., Shao, T., Periyakoil, P., Tipton, C., Colasacco, C., Serpedin, A., Elemento, O., Sabuncu, M., et al. (2025) · The Laryngoscope · doi:10.1002/lary.32036 · Cited by 9
“22 To generate independent test and train data splits, a strati-fi ed shuf fl e split sampling scheme was used.”
Celepli, S., Bigat, I., Karakas, B., Tezcan, H. M., Yar, M. D., Celepli, P., Aksahin, M. F., Hancerliogullari, O., Yilmaz, Y. F., & Erogul, O. (2025) · Diagnostics · doi:10.3390/diagnostics15162065 · Cited by 6
Ma, S., Liao, W., Zhang, Y., Zhang, F., Wang, Y., Lu, Z., Zhao, C., Yu, J., & He, P. (2025) · BioMedical Engineering OnLine · doi:10.1186/s12938-025-01401-9 · Cited by 6
“Prior studies have demonstrated that using intensity-invariant feature extraction methods can maintain high classification accuracy despite such variability [37].”
Metadata
- Authors
- Low, D. M., Rao, V., Randolph, G., Song, P. C., & Ghosh, S. S.
- Affiliations
- Harvard University (US)
- Massachusetts Eye and Ear Infirmary (US)
- Corresponding author
- Low, Daniel M.
- Venue
- PLOS Digital Health
- Publisher
- Public Library of Science (PLoS)
- Published
- 2024-05-30
- Volume
- 3
- Issue
- 5
- Pages
- e0000516
- ISSN
- 2767-3170 (ISSN-L)
- PMC
- PMC11139298
- OpenAlex
- W4399169406
- Cited by
- 16
- References
- 65
- Citations by year
- 2026: 6 · 2025: 6 · 2024: 4
- FWCI
- 4.55
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLOS Digital Health (publishedVersion, cc-by)
- medRxiv (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (submittedVersion, cc-by-nc-nd)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Institute on Deafness and Other Communication Disorders (5T32DC000038-28)
- RallyPoint Fellowship
- Amelia Peabody Charitable Fund (Professional Development Award)
- Gift to McGovern Institute for Brain Research at MIT
- National Institute of Biomedical Imaging and Bioengineering (R01 EB020740, P41 EB019936)
- NIH Office of the Director (1OT2OD032720-01)
- Topics
- Health Sciences › Health Professions › Speech and Hearing › Voice and Speech Disorders
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Pulmonary and Respiratory Medicine › Phonocardiography and Auscultation Techniques
- Keywords
- unilateral vocal fold paralysis, voice disorders, voice screening, acoustic voice analysis, explainable machine learning, SHAP, clinician ratings, data bias, recording duration, vocal intensity, laryngology, remote screening
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Identifying bias in models that detect vocal fold paralysis from audio recordings using explainable machine learning and clinician ratings. https://app.cochat.ai/papers/10.1371/journal.pdig.0000516
- The paper
- Low, D. M., Rao, V., Randolph, G., Song, P. C., & Ghosh, S. S. (2024). Identifying bias in models that detect vocal fold paralysis from audio recordings using explainable machine learning and clinician ratings. PLOS Digital Health. https://doi.org/10.1371/journal.pdig.0000516
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.