CoChat
Sign Up
Cited by 47Voice and Speech DisordersEmotion and Mood RecognitionDementia and Cognitive Impairment Research

Repeatability of Commonly Used Speech and Language Features for Clinical Applications

Open access
Stegmann, Gabriela M., Hahn, Shira, Liss, Julie, Shefner, Jeremy, Rutkove, Seward B., Kawabata, Kan, Bhandari, Samarth, Shelton, Kerisa +2 more·2020·Digital Biomarkers
DOI 10.1159/000511671PMC PMC7772887
Open this paper in CoChat — free accountOpen PDF

Abstract

INTRODUCTION: Changes in speech have the potential to provide important information on the diagnosis and progression of various neurological diseases. Many researchers have relied on open-source speech features to develop algorithms for measuring speech changes in clinical populations as they are convenient and easy to use. However, the repeatability of open-source features in the context of neurological diseases has not been studied. METHODS: We used a longitudinal sample of healthy controls, individuals with amyotrophic lateral sclerosis, and individuals with suspected frontotemporal dementia, and we evaluated the repeatability of acoustic and language features separately on these 3 data sets. RESULTS: Repeatability was evaluated using intraclass correlation (ICC) and the within-subjects coefficient of variation (WSCV). In 3 sets of tasks, the median ICC were between 0.02 and 0.55, and the median WSCV were between 29 and 79%. CONCLUSION: Our results demonstrate that the repeatability of speech features extracted using open-source tool kits is low. Researchers should exercise caution when developing digital health models with open-source speech features. We provide a detailed summary of feature-by-feature repeatability results (ICC, WSCV, SE of measurement, limits of agreement for WSCV, and minimal detectable change) in the online supplementary material so that researchers may incorporate repeatability information into the models they develop.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1159/000511671

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Any non-AI or AI speech or language task should be robust to variation within short time periods when that variation would not be due to the speech difference itself.

    “To demonstrate consistency over repeated measurements, any non-AI or AI speech or language task should be robust to variation within short time periods when that variation would not be due to the speech difference itself (Stegmann et al., 2020).”

    From Equipping Speech-Language Clinicians for the Critical Appraisal of an Artificial Intelligence–Driven, Evidence-Based Future (2025) · cites this paper as Stegmann et al., 2020

    The Introduction (page unavailable) defines repeatability as variation over a short period when disease-related decline is unlikely, and explains why natural variation must be distinguished from disease-related change. The Sources of Speech Variation and Mitigation Strategies section (page unavailable) calls for clinical features robust to nuisance variables. The paper supports that principle for speech and language features used in clinical applications, but does not extend it to every non-AI or AI task. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 27

  1. Ranjan Y, Rashid Z, Stewart C, Conde P, Begale M, Verbeeck D, et al.; RADAR-CNS Consortium. The Hyve, Dobson R, Folarin A, The RADAR-CNS Consortium. RADAR-Base: open source mobile health platform for collecting, monitoring, and analyzing data using sensors, wearables, and mobile devices. JMIR Mhealth Uhealth. 2019 Aug; 7(8):e11734.
  2. Narendra NP, Alku P. Dysarthric speech classification from coded telephone speech using glottal features. Speech Commun. 2019; 110: 47-55.
  3. Taguchi T, Tachikawa H, Nemoto K, Suzuki M, Nagano T, Tachibana R, et al. Major depressive disorder discrim- ination using vocal acoustic features. J Affect Disord. 2018 Jan; 225: 214-20.
  4. Norel R, Pietrowicz M, Agurto C, Rishoni S, Cecchi G. Detection of amyotrophic lateral sclerosis (ALS) via acoustic analysis. Proc Interspeech. 2018: 377-81.
  5. Bayestehtashk A, Asgari M, Shafran I, McNames J. Fully automated assessment of the severity of Parkinson's disease from speech. Comput Speech Lang. 2015 Jan; 29(1): 172-85.
  6. Faurholt-Jepsen M, Busk J, Frost M, Vinberg M, Christensen EM, Winther O, et al. Voice analysis as an objective state marker in bipolar disorder. Transl Psychiatry. 2016 Jul; 6(7):e856.
  7. Bland M. An Introduction to Medical Statistics. United Kingdom: Oxford University Press; 2015.
  8. Eyben F, Wöllmer M, Schuller B. OpenSMILE: the Munich versatile and fast open-source audio feature extractor. Proceedings of the International Conference on Multimedia -MM '10; 2010 Oct 25-29. Florence, Italy. New York: ACM Press; 2010. p. 1459-62.
  9. Boersma P. Praat, a system for doing phonetics by computer. Glot Int. 2001; 5: 341-5.
  10. Eyben F, Scherer KR, Schuller BW, Sundberg J, Andre E, Busso C. The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing. IEEE Trans Affect Comput. 2016; 7(2): 190-202.
  11. Li L, Zeng L, Lin ZJ, Cazzell M, Liu H. Tutorial on use of intraclass correlation coefficients for assessing intertest reliability and its application in functional near-infrared spectroscopy-based brain imaging. J Biomed Opt. 2015 May; 20(5): 50801.
  12. Rousson V, Gasser T, Seifert B. Assessing intrarater, interrater and test-retest reliability of continuous measurements. Stat Med. 2002 Nov; 21(22): 3431-46.
  13. Bates D, Mächler M, Bolker B, Walker S. Fitting linear mixed-effects models using lme4. J Stat Softw. 2015; 67(1): 1-48.
  14. Chiaramonte R, Bonfiglio M. Acoustic analysis of voice in bulbar amyotrophic lateral sclerosis: a systematic review and meta-analysis of studies. Logoped Phoniatr Vocol. 2019 Nov; 1-13.
  15. Vieira H, Costa N, Sousa T, Reis S, Coelho L. Voice-based classification of amyotrophic lateral sclerosis: where are we and where are we going? A systematic review. Neurodegener Dis. 2019; 19(5-6): 163-70.
  16. Bang YI, Min K, Sohn YH, Cho SR. Acoustic characteristics of vowel sounds in patients with Parkinson disease. NeuroRehabilitation. 2013; 32(3): 649-54.
  17. Fleiss JL. The Design and Analysis of Clinical Experiments. Wiley; 1999.
  18. Eyben F, Huber B, Marchi E, Schuller D, Schuller B. Real-time robust recognition of speakers' emotions and characteristics on mobile platforms. In: 2015 International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE; 2015: 778-80.
  19. Lu X. The Relationship of Lexical Richness to the Quality of ESL Learners' Oral Narratives. Mod Lang J. 2012; 96(2): 190-208.
  20. Sherman EM, Brooks BL, Iverson GL, Slick DJ, Strauss E. Reliability and validity in neuropsychology. The little black book of neuropsychology. Boston (MA): Springer; 2011. pp. 873-92.
  21. D'Alatri L, Paludetti G, Contarino MF, Galla S, Marchese MR, Bentivoglio AR. Effects of bilateral subthalamic nucleus stimulation and medication on parkinsonian speech impairment. J Voice. 2008 May; 22(3): 365-72.
  22. Schuller B, Steidl S, Batliner A, Hantke S, Honig F, Orozco-Arroyave JR, et al. The INTERSPEECH 2015 compu- tational paralinguistics challenge: Nativeness, Parkinson's & eating condition. In Proceedings of INTER- SPEECH 2015. 2015; 478-482.
  23. Ringeval F, Schuller B, Valstar M, Cummins N, Cowie R, Tavabi L, et al. AVEC 2019 workshop and challenge: State-of-mind, detecting depression with AI, and cross-cultural affect recognition. In Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop. 2019; 3-12.
  24. Valstar M, Pantic M, Gratch J, Schuller B, Ringeval F, Lalanne D, et al. AVEC 2016: Depression, Mood, and Emotion Recognition Workshop and Challenge. Proceedings of the 6th International Workshop on Audio/ Visual Emotion Challenge -AVEC '16; 2016 Oct; Amsterdam, The Netherlands. New York; Amsterdam; 2016. p. 3-10.
  25. Ng AY. Preventing overfitting of cross-validation data. In Proceedings of the Fourteenth International Conference on Machine Learning. 1997.
  26. Akhtar N, Mian A. Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access. 2018; 6: 14410-30.
  27. Vabalas A, Gowen E, Poliakoff E, Casson AJ. Machine learning algorithm validation with a limited sample size. PLoS One. 2019 Nov 7; 14(11):e0224365.

Cited by

Sorted by citations

Authors
Stegmann, G. M., Hahn, S., Liss, J., Shefner, J., Rutkove, S. B., Kawabata, K., Bhandari, S., Shelton, K., Duncan, C. J., & Berisha, V.
Affiliations
  • Arizona State University (US)
  • Barrow Neurological Institute (US)
  • Beth Israel Deaconess Medical Center (US)
ORCID
Venue
Digital Biomarkers
Publisher
S. Karger AG
Published
2020-12-02
Volume
4
Issue
3
Pages
109-122
ISSN
2504-110X (ISSN-L)
DOI
10.1159/000511671
PMC
PMC7772887
OpenAlex
W3110175868
Cited by
47
References
3
Citations by year
2026: 4 · 2025: 5 · 2024: 11 · 2023: 10 · 2022: 11 · 2021: 6
FWCI
4.06
Type
journal article
Language
en
Open access
Licence
Locations
  • Digital Biomarkers (publishedVersion, cc-by-nc)
  • Europe PMC (PubMed Central) (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Science Foundation (1853247)
  • National Institutes of Health (NIH-R01)
  • National Institute on Deafness and Other Communication Disorders (R43DC017625, R01DC006859)
Topics
  • Health Sciences › Medicine › Physiology › Voice and Speech Disorders
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Health Sciences › Medicine › Psychiatry and Mental health › Dementia and Cognitive Impairment Research
Keywords
acoustic features, linguistic features, amyotrophic lateral sclerosis, frontotemporal dementia, neurological disease diagnosis, minimal detectable change, limits of agreement

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review