Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.

Video-Based Activity Recognition for Automated Motor Assessment of Parkinson's Disease

Open access
Sarapata, Grzegorz, Dushin, Yuriy, Morinan, Gareth, Ong, Joshua, Budhdeo, Sanjay, Kainz, Bernhard, O'Keeffe, Jonathan · 2023 · IEEE Journal of Biomedical and Health Informatics

Published version of a preprint

Abstract

Over the last decade, video-enabled mobile devices have become ubiquitous, while advances in markerless pose estimation allow an individual's body position to be tracked accurately and efficiently across the frames of a video. Previous work by this and other groups has shown that pose-extracted kinematic features can be used to reliably measure motor impairment in Parkinson's disease (PD). This presents the prospect of developing an asynchronous and scalable, video-based assessment of motor dysfunction. Crucial to this endeavour is the ability to automatically recognise the class of an action being performed, without which manual labelling is required. Representing the evolution of body joint locations as a spatio-temporal graph, we implement a deep-learning model for video and frame-level classification of activities performed according to part 3 of the Movement Disorder Society Unified PD Rating Scale (MDS-UPDRS). We train and validate this system using a dataset of n = 7310 video clips, recorded at 5 independent sites. This approach reaches human-level performance in detecting and classifying periods of activity within monocular video clips. Our framework could support clinical workflows and patient care at scale through applications such as quality monitoring of clinical data collection, automated labelling of video streams, or a module within a remote self-assessment system.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/jbhi.2023.3298530

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Automated motor assessment pipeline. A) The MDS-UPDRS motor assessment is recorded, for example, using a mobile device such as a tablet and a collection of videos and corresponding severity scores is saved. B) A pose estimation library OpenPose is used to estimate the patient's joint locations in each video frame and a spatio-temporal graph representation of the patient's movement is created. C) The pose graph is processed using a deep-learning model which outputs a vector of predicted activity confidence scores for each frame of the input. D) Regions of patient activity are identified based on the frame-level prediction output. E) Activity-specific joint-based signal is extracted using the predicted region of activity and forms an input to a disease severity estimation model [10]-[12]
Fig. 1 Automated motor assessment pipeline. A) The MDS-UPDRS motor assessment is recorded, for example, using a mobile device such as a tablet and a collection of videos and corresponding severity scores is saved. B) A pose estimation library OpenPose is used to estimate the patient's joint locations in each video frame and a spatio-temporal graph representation of the patient's movement is created. C) The pose graph is processed using a deep-learning model which outputs a vector of predicted activity confidence scores for each frame of the input. D) Regions of patient activity are identified based on the frame-level prediction output. E) Activity-specific joint-based signal is extracted using the predicted region of activity and forms an input to a disease severity estimation model [10]-[12]CC-BY · From the PDF
Fig. 2 Model-predicted ROI extraction strategy. Raw frame-level output of the ST-GCN network (top). Connected regions of similar activities are grouped together (middle). Final prediction output after frame majority voting over grouped regions (bottom).
Fig. 2 Model-predicted ROI extraction strategy. Raw frame-level output of the ST-GCN network (top). Connected regions of similar activities are grouped together (middle). Final prediction output after frame majority voting over grouped regions (bottom).CC-BY · From the PDF
Fig. 3 Row-normalised confusion matrix for video-level classification of MDS-UPDRS related activities. The ST-GCN achieves almost perfect prediction accuracy regardless of the target item. Most prominent confusion arises within groups of hand- and leg-related items, which could be explained by the high similarity of poses involved.
Fig. 3 Row-normalised confusion matrix for video-level classification of MDS-UPDRS related activities. The ST-GCN achieves almost perfect prediction accuracy regardless of the target item. Most prominent confusion arises within groups of hand- and leg-related items, which could be explained by the high similarity of poses involved.CC-BY · From the PDF
Fig. 4 Examples of human- and model-annotated regions of activity. Video 1) The model has been able to identify the activity in the very first frame of the video despite the absence of prior context. Video 2) The predicted region of Postural Tremor of Hands extends significantly beyond the human annotation. The way PT activity is performed can vary between assessment sites, leading to inconsistency in labelling. Video 3) Regions of correct activity are accurately localised within the video. Despite the relatively low IOU score for Gait Towards Camera class, the automatically annotated region exceeds the human label by less than a second on either end.
Fig. 4 Examples of human- and model-annotated regions of activity. Video 1) The model has been able to identify the activity in the very first frame of the video despite the absence of prior context. Video 2) The predicted region of Postural Tremor of Hands extends significantly beyond the human annotation. The way PT activity is performed can vary between assessment sites, leading to inconsistency in labelling. Video 3) Regions of correct activity are accurately localised within the video. Despite the relatively low IOU score for Gait Towards Camera class, the automatically annotated region exceeds the human label by less than a second on either end.CC-BY · From the PDF
Fig. 5 ROI prediction performance for different patient groups. The performance decreases for patients with higher PD severity. The severity score of 4 is often assigned to patients who are unable to perform the desired activity. A lack of kinetic information can lead to model failure, as it cannot see the subject's intention to perform the task. The model displays no significant difference within sex and age groups. Age grouping has been based on the median age. Standard deviations of 5-fold cross-validation results were used to produce the presented error bars.
Fig. 5 ROI prediction performance for different patient groups. The performance decreases for patients with higher PD severity. The severity score of 4 is often assigned to patients who are unable to perform the desired activity. A lack of kinetic information can lead to model failure, as it cannot see the subject's intention to perform the task. The model displays no significant difference within sex and age groups. Age grouping has been based on the median age. Standard deviations of 5-fold cross-validation results were used to produce the presented error bars.CC-BY · From the PDF
Fig. 6 Confusion between No Action frames and predicted classes, grouped by MDS-UPDRS items. The No Action frames are mainly confused with the video item label, which suggests that regions of true activity are over-predicted by the model, or, more likely in our view, have not been annotated.
Fig. 6 Confusion between No Action frames and predicted classes, grouped by MDS-UPDRS items. The No Action frames are mainly confused with the video item label, which suggests that regions of true activity are over-predicted by the model, or, more likely in our view, have not been annotated.CC-BY · From the PDF
Fig. 7 Comparison of human and model performance in identifying 'robust' regions of patient activity. For half of the item classes, the model-human agreement level matches that of human-human. 95% confidence intervals presented in the figure are based on 1000 bootstrap iterations. Annotation of significant differences is based on a paired t-test with Bonferroni correction applied to the resulting p-values.
Fig. 7 Comparison of human and model performance in identifying 'robust' regions of patient activity. For half of the item classes, the model-human agreement level matches that of human-human. 95% confidence intervals presented in the figure are based on 1000 bootstrap iterations. Annotation of significant differences is based on a paired t-test with Bonferroni correction applied to the resulting p-values.CC-BY · From the PDF
Image 8
CC-BY · From the PDF
Image 9
CC-BY · From the PDF
Image 10
CC-BY · From the PDF
Fig. 8 Row-normalised confusion matrices of disease severity estimates based on human-annotated ROIs (Top-left) and model-annotated ROIs (Top-right). The difference in severity score estimation accuracy based on human- and model-annotated regions of activity with 95% bootstrap confidence intervals (Bottom). In summary, for most clinical scores the severity estimation models display no systematic differences in performance when model-annotated ROIs and associated kinetic features are used to infer the disease severity.
Fig. 8 Row-normalised confusion matrices of disease severity estimates based on human-annotated ROIs (Top-left) and model-annotated ROIs (Top-right). The difference in severity score estimation accuracy based on human- and model-annotated regions of activity with 95% bootstrap confidence intervals (Bottom). In summary, for most clinical scores the severity estimation models display no systematic differences in performance when model-annotated ROIs and associated kinetic features are used to infer the disease severity.CC-BY · From the PDF
Fig. 9 Example implementation of continuous activity recognition with raw (no post-processing) frame-level prediction. The model has been able to identify multiple activity classes in a single video, with overall frame prediction accuracy of 91 . 8 %. Video available at https://youtu.be/ 7dqgAHdJank
Fig. 9 Example implementation of continuous activity recognition with raw (no post-processing) frame-level prediction. The model has been able to identify multiple activity classes in a single video, with overall frame prediction accuracy of 91 . 8 %. Video available at https://youtu.be/ 7dqgAHdJankCC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 34 references

Cited by

See all 49

Metadata

Authors
Sarapata, G., Dushin, Y., Morinan, G., Ong, J., Budhdeo, S., Kainz, B., & O'Keeffe, J.
Affiliations
  • Machine Medicine Technologies, London, U.K.
  • UCL Queen Square Institute of Neurology (GB)
  • Imperial College London (GB)
Venue
IEEE Journal of Biomedical and Health Informatics
Published
2023-10
Volume
27
Issue
10
Pages
5032-5041
ISSN
  • 2168-2194 (ISSN-L)
  • 2168-2208
OpenAlex
W4385246386
Cited by
49
References
34
Citations by year
2026: 9 · 2025: 23 · 2024: 14 · 2023: 3
FWCI
4.45
Type
journal article
Language
en
Open access
Locations
  • IEEE Journal of Biomedical and Health Informatics (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • UCL Discovery (University College London) (submittedVersion, cc-by)
  • Figshare (submittedVersion, cc-by)
  • Spiral (Imperial College London) (submittedVersion, cc-by)
Funders
Machine Medicine Technologies
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
  • Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
Keywords
Parkinson's disease, motor function assessment, activity recognition, video-based assessment, markerless pose estimation, spatio-temporal graph, deep learning, motor impairment
MeSH terms
Mental Status and Dementia Tests, Biomechanical Phenomena, Humans, Parkinson Disease, Severity of Illness Index

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Video-Based Activity Recognition for Automated Motor Assessment of Parkinson's Disease. https://app.cochat.ai/papers/10.1109/jbhi.2023.3298530
The paper
Sarapata, G., Dushin, Y., Morinan, G., Ong, J., Budhdeo, S., Kainz, B., & O'Keeffe, J. (2023). Video-Based Activity Recognition for Automated Motor Assessment of Parkinson's Disease. IEEE Journal of Biomedical and Health Informatics. https://doi.org/10.1109/jbhi.2023.3298530

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.