Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.

A Human-AI Collaborative Approach for Clinical Decision Making on Rehabilitation Assessment

Open access
Lee, Min Hun, Siewiorek, Daniel P., Smailagic, Asim, Bernardino, Alexandre, Bermúdez i Badia, Sergi Bermúdez · 2021 · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Abstract

Advances in artificial intelligence (AI) have made it increasingly applicable to supplement expert’s decision-making in the form of a decision support system on various tasks. For instance, an AI-based system can provide therapists quantitative analysis on patient’s status to improve practices of rehabilitation assessment. However, there is limited knowledge on the potential of these systems. In this paper, we present the development and evaluation of an interactive AI-based system that supports collaborative decision making with therapists for rehabilitation assessment. This system automatically identifies salient features of assessment to generate patient-specific analysis for therapists, and tunes with their feedback. In two evaluations with therapists, we found that our system supports therapists significantly higher agreement on assessment (0.71 average F1-score) than a traditional system without analysis (0.66 average F1-score, p < 0.05). After tuning with therapist’s feedback, our system significantly improves its performance from 0.8377 to 0.9116 average F1-scores (p < 0.01). This work discusses the potential of a human-AI collaborative system to support more accurate decision making while learning from each other’s strengths.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3411764.3445472

Cites

62 of 62 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Flow diagram of an interactive approach for human and artificial intelligence (AI) collaborative decision making on rehabilitation assessment: an AI-based system automatically selects kinematic features of assessment to predict the quality of motion and generate patient-specific analysis on the visualization interface. A therapist can review this summarized, patient-specific analysis to improve understanding on patient's performance and provide feature-based feedback to tune a model for personalized assessment
Figure 1 Flow diagram of an interactive approach for human and artificial intelligence (AI) collaborative decision making on rehabilitation assessment: an AI-based system automatically selects kinematic features of assessment to predict the quality of motion and generate patient-specific analysis on the visualization interface. A therapist can review this summarized, patient-specific analysis to improve understanding on patient's performance and provide feature-based feedback to tune a model for personalized assessmentCC-BY · From the PDF
Figure 2 The visualization interface of the proposed system that presents (a) the video of patient's exercise motions and the predicted quality of motion with (b) overall feature analysis with three most important features, (c) detailed feature values with a specified threshold value of feature for assessment, and (d) trajectory trends between unaffected and affected side.
Figure 2 The visualization interface of the proposed system that presents (a) the video of patient's exercise motions and the predicted quality of motion with (b) overall feature analysis with three most important features, (c) detailed feature values with a specified threshold value of feature for assessment, and (d) trajectory trends between unaffected and affected side.CC-BY · From the PDF
Figure 3 Performance (average F1-scores) of various machine learning (ML) models, non-interactive, initial rulebased and hybrid models (RB 1 and HM 1) and therapists' agreement.
Figure 3 Performance (average F1-scores) of various machine learning (ML) models, non-interactive, initial rulebased and hybrid models (RB 1 and HM 1) and therapists' agreement.CC-BY · From the PDF
Figure 4 Results of the User Study in term of (a) Responses of Questionnaires and (b) Agreement Level of Therapists' Assessments: the proposed system is more useful, richer, more trustful while reducing effort and frustration on assessment tasks. It is more likely to be used in the clinical practices than other baseline systems (i.e. traditional and predscore), and achieves the highest agreement on therapists' assessment. We indicate statistical significance using one-way ANOVA tests at the bottom of each variable and post-hoc, paired t-tests at the top of each variable. *, **, and *** indicate statistical significance using the one-way ANOVA tests or paired t-tests at 90%, 95%, and 99% significance level.
Figure 4 Results of the User Study in term of (a) Responses of Questionnaires and (b) Agreement Level of Therapists' Assessments: the proposed system is more useful, richer, more trustful while reducing effort and frustration on assessment tasks. It is more likely to be used in the clinical practices than other baseline systems (i.e. traditional and predscore), and achieves the highest agreement on therapists' assessment. We indicate statistical significance using one-way ANOVA tests at the bottom of each variable and post-hoc, paired t-tests at the top of each variable. *, **, and *** indicate statistical significance using the one-way ANOVA tests or paired t-tests at 90%, 95%, and 99% significance level.CC-BY · From the PDF
Figure 5 The effect of therapist's feature-based feedback: (a) the performance of models over feedback, iterations and (b) the comparison of model performance without/with feature-based feedback. Both rule-based and hybrid models significantly improve their performance with therapist's feature-based feedback ( p < 0 . 01 using paired t-tests). Interactive, tuned hybrid model (HM 10) performs better than the machine learning model with neural networks (ML-NN) and therapists' agreement.
Figure 5 The effect of therapist's feature-based feedback: (a) the performance of models over feedback, iterations and (b) the comparison of model performance without/with feature-based feedback. Both rule-based and hybrid models significantly improve their performance with therapist's feature-based feedback ( p < 0 . 01 using paired t-tests). Interactive, tuned hybrid model (HM 10) performs better than the machine learning model with neural networks (ML-NN) and therapists' agreement.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 62 references

Research works printed but not linked: 6

  1. Mark T Bayley, Amanda Hurdowar, Carol L Richards, Nicol Korner-Bitensky, Sharon Wood-Dauphinee, Janice J Eng, Marilyn McKay-Lyons, Edward Harrison, Robert Teasell, Margaret Harrison, 2012. Barriers to implementation of stroke rehabilitation evidence: findings from a multi-site pilot project. Disability and rehabilitation 34, 19 (2012), 1633–1638.
  2. Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In IJCAI-17 workshop on explainable AI (XAI), Vol. 8. 1.
  3. Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. 2002. Gene selection for cancer classification using support vector machines. Machine learning 46, 1-3 (2002), 389–422.
  4. Margit Alt Murphy, Carin Willén, and Katharina S Sunnerhagen. 2011. Kinematic variables quantifying upper-extremity performance after stroke during reaching and drinking from a glass. Neurorehabilitation and neural repair 25, 1 (2011), 71–80.
  5. Jiliang Tang, Salem Alelyani, and Huan Liu. 2014. Feature selection for classification: A review. Data classification: Algorithms and applications (2014), 37.
  6. Eric J Topol. 2019. High-performance medicine: the convergence of human and artificial intelligence. Nature medicine 25, 1 (2019), 44–56.

Cited by

See all 164

Metadata

Authors
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. B.
Affiliations
  • Carnegie Mellon University (US)
  • University of Lisbon (PT)
  • Universidade da Madeira / Madeira-ITI, NOVA-LINCS, Portugal
Venue
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2021-05-06
Pages
1-14
OpenAlex
W3161926791
Cited by
137
References
62
Citations by year
2026: 14 · 2025: 47 · 2024: 29 · 2023: 31 · 2022: 15 · 2021: 1
FWCI
17.03
Type
conference paper
Language
en
Open access
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion)
  • Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, other-oa)
  • Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) (submittedVersion, cc-by)
Funders
  • ERDF through the LISBOA 2020 and the FCT (LISBOA-01-0247-FEDER-045948)
  • National Science Foundation (CNS-1518865)
  • FCT LARSyS - Plurianual funding 2020-2023 (UIDB/50009/2020)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Technology and Human Factors in Education and Health
Keywords
human-AI collaboration, clinical decision support, rehabilitation assessment, therapist feedback, patient-specific analysis, adaptive decision support system

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Human-AI Collaborative Approach for Clinical Decision Making on Rehabilitation Assessment. https://app.cochat.ai/papers/10.1145/3411764.3445472
The paper
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. B. (2021). A Human-AI Collaborative Approach for Clinical Decision Making on Rehabilitation Assessment. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445472

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.