CoChat

Towards Efficient Annotations for a Human-AI Collaborative, Clinical Decision Support System: A Case Study on Physical Stroke Rehabilitation Assessment

Open access
Lee, Min Hun, Siewiorek, Daniel P., Smailagic, Asim, Bernardino, Alexandre, Bermúdez i Badia, Sergi · 2022 · 27th International Conference on Intelligent User Interfaces

The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.

Abstract

Artificial intelligence (AI) and machine learning (ML) algorithms are increasingly being explored to support various decision-making tasks in health (e.g. rehabilitation assessment). However, the development of such AI/ML-based decision support systems is challenging due to the expensive process to collect an annotated dataset. In this paper, we describe the development process of a human-AI collaborative, clinical decision support system that augments an ML model with a rule-based (RB) model from domain experts. We conducted its empirical evaluation in the context of assessing physical stroke rehabilitation with the dataset of three exercises from 15 post-stroke survivors and therapists. Our results bring new insights on the efficient development and annotations of a decision support system: when an annotated dataset is not available initially, the RB model can be used to assess post-stroke survivor’s quality of motion and identify samples with low confidence scores to support efficient annotations for training an ML model. Specifically, our system requires only 22 - 33% of annotations from therapists to train an ML model that achieves equally good performance with an ML model with all annotations from a therapist. Our work discusses the values of a human-AI collaborative approach for effectively collecting an annotated dataset and supporting a complex decision-making task.

Overview

Integrity signals

Registry

Crossref · 10.1145/3490099.3511112

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 9

  1. Mark T Bayley , Amanda Hurdowar , Carol L Richards , Nicol Korner-Bitensky , Sharon Wood-Dauphinee , Janice J Eng , Marilyn McKay-Lyons, Edward Harrison , Robert Teasell, Margaret Harrison , 2012 . Barriers to implementation of stroke rehabilitation evidence: findings from a multi-site pilot project. Disability and rehabilitation 34, 19 (2012), 1633–1638. Mark T Bayley, Amanda Hurdowar, Carol L Richards, Nicol Korner-Bitensky, Sharon Wood-Dauphinee, Janice J Eng, Marilyn McKay-Lyons, Edward Harrison, Robert Teasell, Margaret Harrison, 2012. Barriers to implementation of stroke rehabilitation evidence: findings from a multi-site pilot project. Disability and rehabilitation 34, 19 (2012), 1633–1638.
  2. Or Biran and Courtenay Cotton . 2017 . Explanation and justification in machine learning: A survey . In IJCAI-17 workshop on explainable AI (XAI), Vol. 8. 1. Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In IJCAI-17 workshop on explainable AI (XAI), Vol. 8. 1.
  3. Po- Hsuan Cameron Chen , Yun Liu , and Lily Peng . 2019. How to develop machine learning models for healthcare. Nature materials 18, 5 ( 2019 ), 410. Po-Hsuan Cameron Chen, Yun Liu, and Lily Peng. 2019. How to develop machine learning models for healthcare. Nature materials 18, 5 (2019), 410.
  4. Andre Esteva , Brett Kuprel , Roberto A Novoa , Justin Ko , Susan M Swetter , Helen M Blau , and Sebastian Thrun . 2017. Dermatologist-level classification of skin cancer with deep neural networks. nature 542, 7639 ( 2017 ), 115–118. Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. 2017. Dermatologist-level classification of skin cancer with deep neural networks. nature 542, 7639 (2017), 115–118.
  5. Chuan Guo , Geoff Pleiss , Yu Sun , and Kilian Q Weinberger . 2017 . On calibration of modern neural networks . In International Conference on Machine Learning. PMLR, 1321–1330 . Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International Conference on Machine Learning. PMLR, 1321–1330.
  6. Saif Khairat , David Marc , William Crosby , and Ali Al Sanousi . 2018. Reasons for physicians not adopting clinical decision support systems: critical analysis. JMIR medical informatics 6, 2 ( 2018 ), e24. Saif Khairat, David Marc, William Crosby, and Ali Al Sanousi. 2018. Reasons for physicians not adopting clinical decision support systems: critical analysis. JMIR medical informatics 6, 2 (2018), e24.
  7. Been Kim Julie A Shah and Finale Doshi-Velez. 2015. Mind the gap: A generative approach to interpretable feature selection and extraction. In Advances in Neural Information Processing Systems. 2260–2268. Been Kim Julie A Shah and Finale Doshi-Velez. 2015. Mind the gap: A generative approach to interpretable feature selection and extraction. In Advances in Neural Information Processing Systems. 2260–2268.
  8. Jon Kleinberg , Himabindu Lakkaraju , Jure Leskovec , Jens Ludwig , and Sendhil Mullainathan . 2018. Human decisions and machine predictions. The quarterly journal of economics 133, 1 ( 2018 ), 237–293. Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 2018. Human decisions and machine predictions. The quarterly journal of economics 133, 1 (2018), 237–293.
  9. Edward Taub , David M Morris , Jean Crago , Danna Kay King , Mary Bowman , Camille Bryson , Staci Bishop , Sonya Pearson , and Sharon E Shaw . 2011. Wolf motor function test (WMFT) manual. Birmingham: University of Alabama , CI Therapy Research Group ( 2011 ). Edward Taub, David M Morris, Jean Crago, Danna Kay King, Mary Bowman, Camille Bryson, Staci Bishop, Sonya Pearson, and Sharon E Shaw. 2011. Wolf motor function test (WMFT) manual. Birmingham: University of Alabama, CI Therapy Research Group (2011).

Cited by

See all 19

Metadata

Authors
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S.
Affiliations
  • Singapore Management University (SG)
  • Carnegie Mellon University (US)
  • University of Lisbon (PT)
  • Faculdade de Ciências Exatas e da Engenharia, Universidade da Madeira, Portugal
Venue
27th International Conference on Intelligent User Interfaces
Publisher
ACM
Published
2022-03-22
Pages
4-14
OpenAlex
W4220855893
Cited by
19
References
48
Citations by year
2026: 5 · 2025: 6 · 2024: 4 · 2023: 4
FWCI
12.48
Type
conference paper
Language
en
Open access
Locations
  • ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion, cc-by-nc-nd)
  • Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) (publishedVersion)
  • Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, other-oa)
Funders
  • LISBOA 2020 and the FCT (LISBOA-01-0247-FEDER-045948)
  • Singapore Ministry of Education (MOE) (21-SIS-SMU-034)
  • National Science Foundation (CNS-1518865)
  • FCT (UIDB/50009/2020)
Topics
  • Health Sciences › Medicine › Rehabilitation › Stroke Rehabilitation and Recovery
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
clinical decision support systems, human-AI collaboration, stroke rehabilitation assessment, physiotherapy, rule-based models, domain expert knowledge

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.