Multimodal assessment of Parkinson’s disease: a deep learning approach
Open accessAbstract
Parkinson's disease is a neurodegenerative disorder characterized by a variety of motor symptoms. Particularly, difficulties to start/stop movements have been observed in patients. From a technical/diagnostic point of view, these movement changes can be assessed by modeling the transitions between voiced and unvoiced segments in speech, the movement when the patient starts or stops a new stroke in handwriting, or the movement when the patient starts or stops the walking process. This study proposes a methodology to model such difficulties to start or to stop movements considering information from speech, handwriting, and gait. We used those transitions to train convolutional neural networks to classify patients and healthy subjects. The neurological state of the patients was also evaluated according to different stages of the disease (initial, intermediate, and advanced). In addition, we evaluated the robustness of the proposed approach when considering speech signals in three different languages: Spanish, German, and Czech. According to the results, the fusion of information from the three modalities is highly accurate to classify patients and healthy subjects, and it shows to be suitable to assess the neurological state of the patients in several stages of the disease. We also aimed to interpret the feature maps obtained from the deep learning architectures with respect to the presence or absence of the disease and the neurological state of the patients. As far as we know, this is one of the first works that considers multimodal information to assess Parkinson's disease following a deep learning approach.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The studies by Lahoti et al and Vasquez-Correa et al used the same dataset (PC-GITA).
“the studies by Lahoti et al [11] and Vasquez-Correa et al [13] used the same dataset (PC-GITA)”
From Explainable and Interpretable AI for Voice and Speech Analysis in Clinical Care: Systematic Review (2026) · cites this paper as [13]
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Goetz, C. G., Tilley, B. C., Shaftman, S. R., Stebbins, G. T., Fahn, S., Martinez‐Martin, P., Poewe, W., Sampaio, C., Stern, M. B., Dodel, R., et al. (2008) · Movement Disorders · doi:10.1002/mds.22340 · Cited by 7,536 · see citing works
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · see citing works
The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing
Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andre, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2457417 · Cited by 1,818 · see citing works
Novel Speech Signal Processing Algorithms for High-Accuracy Classification of Parkinson’s Disease
Tsanas, A., Little, M. A., McSharry, P. E., Spielman, J., & Ramig, L. O. (2012) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2012.2183367 · Cited by 732 · see citing works
Biochemical aspects of Parkinson's disease
Hornykiewicz, O. (1998) · Neurology · doi:10.1212/wnl.51.2_suppl_2.s2 · Cited by 563 · see citing works
Prevalence of Parkinson's disease in Europe : A collaborative study of population-based cohorts
de Rijk, M. C., Launer, L. J., Berger, K., Breteler, M. M. B., Dartigues, J. F., Baldereschi, M., Fratiglioni, L., Lobo, A., Martínez‐Lage, J. M., Trenkwalder, C., et al. (2000) · EUR Research Repository (Erasmus University Rotterdam) · Cited by 514 · see citing works
Rusz, J., Čmejla, R., Ruzickova, H., & Růžička, E. (2011) · The Journal of the Acoustical Society of America · doi:10.1121/1.3514381 · Cited by 498 · see citing works
Evaluation of handwriting kinematics and pressure for differential diagnosis of Parkinson's disease
Drotár, P., Mekyska, J., Rektorová, I., Masárová, L., Smékal, Z., & Faúndez-Zanuy, M. (2016) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2016.01.004 · Cited by 375 · see citing works
Hlavnička, J., Čmejla, R., Tykalová, T., Šonka, K., Růžička, E., & Rusz, J. (2017) · Scientific Reports · doi:10.1038/s41598-017-00047-5 · Cited by 326 · see citing works
Postuma, R. B., Iranzo, Á., Högl, B., Arnulf, I., Ferini‐Strambi, L., Manni, R., Miyamoto, T., Oertel, W. H., Dauvilliers, Y. A., Ju, Y. S., et al. (2015) · Annals of Neurology · doi:10.1002/ana.24385 · Cited by 290 · see citing works
Forrest, K. M., Weismer, G. G., & Turner, G. S. (1989) · The Journal of the Acoustical Society of America · doi:10.1121/1.397755 · Cited by 281 · see citing works
New Spanish speech corpus database for the analysis of people suffering from Parkinson’s disease
Orozco‐Arroyave, J. R., Arias-Londoño, J. D., Vargas‐Bonilla, J. F., Rátiva, M. C. G., & Nöth, E. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3c643i6hcfzo · Cited by 248 · see citing works
Self-citationUnbiased and Mobile Gait Analysis Detects Motor Impairment in Parkinson's Disease
Klucken, J., Barth, J., Kugler, P., Schlachetzki, J. C. M., Henze, T., Marxreiter, F., Kohl, Z., Steidl, R., Joachim Hornegger, A., Eskofier, B. M., et al. (2013) · PLoS ONE · doi:10.1371/journal.pone.0056956 · Cited by 217 · see citing works
Self-citation(2000) · Neurology · doi:10.1212/wnl.54.5.21a · Cited by 209 · see citing works
Pharmacological treatment in Parkinson's disease: Effects on gait
Smulders, K., Dale, M. L., Carlson‐Kuhta, P., Nutt, J. G., & Horak, F. B. (2016) · Parkinsonism & Related Disorders · doi:10.1016/j.parkreldis.2016.07.006 · Cited by 190 · see citing works
Automatic Evaluation of Articulatory Disorders in Parkinson’s Disease
Novotný, M., Rusz, J., Čmejla, R., & Růžička, E. (2014) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2014.2329734 · Cited by 158 · see citing works
Rusz, J., Bonnet, C., Klempíř, J., Tykalová, T., Baborová, E., Novotný, M., Rulseh, A. M., & Růžička, E. (2015) · Journal of Neurology · doi:10.1007/s00415-015-7671-1 · Cited by 148 · see citing works
Canter, G. J. (1965) · Journal of Speech and Hearing Disorders · doi:10.1044/jshd.3003.217 · Cited by 144 · see citing works
García, A. M., Carrillo, F., Orozco-Arroyave, J. R., Trujillo, N., Vargas Bonilla, J. F., Fittipaldi, S., Adolfi, F., Nöth, E., Sigman, M., Fernández Slezak, D., et al. (2016) · Brain and Language · doi:10.1016/j.bandl.2016.07.008 · Cited by 144 · see citing works
Self-citationControl of stroke size, peak acceleration, and stroke duration in Parkinsonian handwriting
Teulings, H.-L. H. M., & Stelmach, G. E. (1991) · Human Movement Science · doi:10.1016/0167-9457(91)90010-u · Cited by 130 · see citing works
Kinematic analysis of dopaminergic effects on skilled handwriting movements in Parkinson’s disease
Tucha, O. M., Mecklinger, L., Thome, J., Reiter, A., Alders, G. L., Sartor, H., Naumann, M., & Lange, K. W. (2005) · Journal of Neural Transmission · doi:10.1007/s00702-005-0346-9 · Cited by 129 · see citing works
NeuroSpeech: An open-source software for Parkinson's speech analysis
Orozco‐Arroyave, J. R., Vásquez-Correa, J. C., Vargas‐Bonilla, J. F., Arora, R., Dehak, N., Nidadavolu, P. S., Christensen, H., Rudzicz, F., Yancheva, M., Chinaei, H., et al. (2017) · Digital Signal Processing · doi:10.1016/j.dsp.2017.07.004 · Cited by 125 · see citing works
Self-citationClinical syndromes: Parkinsonian gait
Ebersbach, G., Moreau, C., Gandor, F., Defebvre, L., & Devos, D. (2013) · Movement Disorders · doi:10.1002/mds.25675 · Cited by 107 · see citing works
Skodda, S., Visser, W., & Schlegel, U. S. (2010) · Journal of Voice · doi:10.1016/j.jvoice.2009.07.005 · Cited by 103 · see citing works
Schuller, B. W., Steidl, S., Batliner, A. M., et al. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-179 · Cited by 101 · see citing works
Patel, S., Chen, B.-R., Buckley, T. A., Rednic, R., McClure, D., Tarsy, D., Shih, L. C., Dy, J., Welsh, M., & Bonato, P. (2010) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/iembs.2010.5627124 · Cited by 93 · see citing works
Tykalová, T., Rusz, J., Klempíř, J., Čmejla, R., & Růžička, E. (2016) · Brain and Language · doi:10.1016/j.bandl.2016.11.005 · Cited by 92 · see citing works
Parisi, F., Ferrari, G., Matteo Giuberti, A., Contin, L., Cimolin, V., Azzaro, C., Albani, G., & Mauro, A. (2015) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2015.2472640 · Cited by 89 · see citing works
Convolutional Neural Network to Model Articulation Impairments in Patients with Parkinson’s Disease
Vásquez-Correa, J. C., Orozco‐Arroyave, J. R., & Nöth, E. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1078 · Cited by 89 · see citing works
Self-citationEyben, F., & Schuller, B. W. (2015) · ACM SIGMultimedia Records · doi:10.1145/2729095.2729097 · Cited by 87 · see citing works
Ghassemi, N. H., Hannink, J., Martindale, C. F., Gaßner, H., Müller, M., Klucken, J., & Eskofier, B. M. (2018) · Sensors · doi:10.3390/s18010145 · Cited by 79 · see citing works
Self-citationRusz, J., Tykalová, T., Klempíř, J., Čmejla, R., & Růžička, E. (2016) · Journal of Neural Transmission · doi:10.1007/s00702-016-1515-8 · Cited by 71 · see citing works
Technologies for Assessment of Motor Disorders in Parkinson’s Disease: A Review
Oung, Q. W., Muthusamy, H., Lee, H. L., Basah, S. N., Yaacob, S., Sarillee, M., & Lee, C. K. (2015) · Sensors · doi:10.3390/s150921710 · Cited by 70 · see citing works
Cerňak, M., Orozco‐Arroyave, J. R., Rudzicz, F., Christensen, H., Vásquez-Correa, J. C., & Nöth, E. (2017) · Computer Speech & Language · doi:10.1016/j.csl.2017.06.004 · Cited by 64 · see citing works
Self-citationA cross-linguistic perspective to the study of dysarthria in Parkinson’s disease
Pinto, S., Chan, A., Guimarães, I., Rothe‐Neves, R., & Sadat, J. (2017) · Journal of Phonetics · doi:10.1016/j.wocn.2017.01.009 · Cited by 63 · see citing works
Alibiglou, L., Videnović, A., Planetta, P. J., Vaillancourt, D. E., & MacKinnon, C. D. (2016) · Movement Disorders · doi:10.1002/mds.26665 · Cited by 47 · see citing works
Barth, J., Sunkel, M., Bergner, K., Schickhuber, G., Winkler, J., Klucken, J., & Eskofier, B. M. (2012) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2012.6347146 · Cited by 44 · see citing works
Self-citationGrósz, T., Busa‐Fekete, R., Gosztolya, G., et al. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-195 · Cited by 42 · see citing works
Multi-view representation learning via gcca for multimodal analysis of Parkinson's disease
Vásquez-Correa, J. C., Orozco‐Arroyave, J. R., Arora, R., Nöth, E., Dehak, N., Christensen, H., Rudzicz, F., Bocklet, T., Cerňak, M., Chinaei, H., et al. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7952700 · Cited by 38 · see citing works
Self-citationInterpretable Objective Assessment of Dysarthric Speech Based on Deep Neural Networks
Tu, M., Berisha, V., & Liss, J. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1222 · Cited by 35 · see citing works
Wearable sensor network for health monitoring: the case of Parkinson disease
Pastorino, M., Arredondo, M. T., Cancela, J., et al. (2013) · Journal of Physics Conference Series · doi:10.1088/1742-6596/450/1/012055 · Cited by 29 · see citing works
Schuller, B., Steidl, S., Batliner, A., et al. (2019) · OPUS (Augsburg University) · Cited by 22 · see citing works
Force Amplitude and Force Duration in Parkinsonian Handwriting
Teulings, H.-L. H. M., & Stelmach, G. E. (1991) · Springer eBooks · doi:10.1007/978-94-011-3626-6_13 · Cited by 15 · see citing works
Hannink, J., Gaßner, H., Winkler, J., Eskofier, B. M., & Klucken, J. (2017) · Basal Ganglia · doi:10.1016/j.baga.2017.02.002 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Multimodal machine learning in precision health: A scoping review
Kline, A., Wang, H., Li, Y., Dennis, S., Hutch, M., Xu, Z., Wang, F., Cheng, F., & Luo, Y. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00712-8 · Cited by 528 · see citing works
Cognitive determinants of dysarthria in Parkinson's disease: An automated machine learning approach
García, A. M., Arias‐Vergara, T., C. Vasquez‐Correa, J., Nöth, E., Schuster, M., Welch, A. E., Bocanegra, Y., Baena, A., & Orozco‐Arroyave, J. R. (2021) · Movement Disorders · doi:10.1002/mds.28751 · Cited by 77 · see citing works
Self-citationThe Effect of the MFCC Frame Length in Automatic Voice Pathology Detection
Tirronen, S., Kadiri, S. R., & Alku, P. (2024) · Journal of Voice · doi:10.1016/j.jvoice.2022.03.021 · Cited by 45 · see citing works
Kodali, M., Kadiri, S. R., & Alku, P. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2023.101548 · Cited by 24 · see citing works
Explainable and Interpretable AI for Voice and Speech Analysis in Clinical Care: Systematic Review
Ebraheem, M., Toghranegar, J., Bridge2AI-Voice Consortium, Bensoussan, Y., & Templeton, J. M. (2026) · Journal of Medical Internet Research · doi:10.2196/83790
- Authors
- Vasquez-Correa, J. C., Arias-Vergara, T., Orozco-Arroyave, J. R., Eskofier, B., Klucken, J., & Noth, E.
- Affiliations
- Friedrich-Alexander-Universität Erlangen-Nürnberg (DE)
- Venue
- IEEE Journal of Biomedical and Health Informatics
- Published
- 2019-07
- Volume
- 23
- Issue
- 4
- Pages
- 1618-1630
- ISSN
- 2168-2194 (ISSN-L)
- 2168-2208
- OpenAlex
- W2913575235
- Cited by
- 262
- References
- 45
- Citations by year
- 2026: 28 · 2025: 50 · 2024: 39 · 2023: 41 · 2022: 37 · 2021: 37 · 2020: 23 · 2019: 7
- FWCI
- 11.56
- Type
- journal article
- Language
- en
- Open access
- green
- zenodo.org
- Licence
- Locations
- IEEE Journal of Biomedical and Health Informatics (publishedVersion)
- PubMed (publishedVersion)
- Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, other-oa)
- Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by)
- Greater South Information System
- Funders
- Deutsche Forschungsgemeinschaft (Heisenberg professorship program (Grant Number ES)
- H2020 Marie Sklodowska-Curie Actions (766287)
- Universidad de Antioquia (CODI grant 2015-7683)
- Departamento Administrativo de Ciencia Tecnologia e Innovacion (Convocatoria Doctorado Nacional-785)
- Topics
- Health Sciences › Medicine › Physiology › Voice and Speech Disorders
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Neurology › Parkinson's Disease Mechanisms and Treatments
- Keywords
- Parkinson's disease, multimodal assessment, speech analysis, handwriting analysis, gait analysis, convolutional neural networks, disease staging, cross-lingual robustness, motor symptoms
- MeSH terms
- Gait Analysis, Deep Learning, Aged, Aged, 80 and over, Female, Gait, Handwriting, Humans, Image Processing, Computer-Assisted, Male, Middle Aged, Parkinson Disease, ROC Curve, Signal Processing, Computer-Assisted, Speech, Databases, Factual
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
