Towards Efficient Annotations for a Human-AI Collaborative, Clinical Decision Support System: A Case Study on Physical Stroke Rehabilitation Assessment
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
Artificial intelligence (AI) and machine learning (ML) algorithms are increasingly being explored to support various decision-making tasks in health (e.g. rehabilitation assessment). However, the development of such AI/ML-based decision support systems is challenging due to the expensive process to collect an annotated dataset. In this paper, we describe the development process of a human-AI collaborative, clinical decision support system that augments an ML model with a rule-based (RB) model from domain experts. We conducted its empirical evaluation in the context of assessing physical stroke rehabilitation with the dataset of three exercises from 15 post-stroke survivors and therapists. Our results bring new insights on the efficient development and annotations of a decision support system: when an annotated dataset is not available initially, the RB model can be used to assess post-stroke survivor’s quality of motion and identify samples with low confidence scores to support efficient annotations for training an ML model. Specifically, our system requires only 22 - 33% of annotations from therapists to train an ML model that achieves equally good performance with an ML model with all annotations from a therapist. Our work discusses the values of a human-AI collaborative approach for effectively collecting an annotated dataset and supporting a complex decision-making task.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“researchers have explored the feasibility of human-AI collaborative decision-making, in which an AI model provides humans new data-driven insights on a task for achieving complementary performance, outperforming neither of the AI or the human alone [5, 30, 42, 43].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [43]
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · cited 2×
“To address this latter issue, researchers have actively explored a method or tool to provide explanations on the predictions of anML-based system [25, 40, 43, 50].”
MethodRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“Although ML-based systems have shown competitive performance, prior work describes the difficulty to integrate these systems into practice due to the lack of human-centered designs as a primary reason for failure [13, 31, 40, 60] and performing as a black-box system [52].”
BackgroundMultimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · cited 2×
“Thus, it would be interesting to further explore whether the HM can support training a better ML model and achieve more efficient annotation after improving the technique of a multimodal machine learning [4] and fine-tuning the RB model.”
BackgroundMethodLanghorne, P., Bernhardt, J., & Kwakkel, G. (2011) · The Lancet · doi:10.1016/s0140-6736(11)60325-5 · Cited by 2,726
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“When this interface presents the predicted assessment, it also includes the confidence score to “make clear how well the system can do” [2].”
BackgroundPredicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884 · cited 1×
“probabilities from the softmax layer of a neural network) [23, 46, 61] and utilize the predicted score of a model, PM (Y |X ) to identify low confidence samples (e.”
MethodIntelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · cited 2×
“Recent advances in artificial intelligence (AI) have made it applicable to support decision-making in healthcare [6, 12, 14, 42].”
BackgroundTransforming classifier scores into accurate multiclass probability estimates
Zadrozny, B., & Elkan, C. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775151 · Cited by 1,280 · cited 1×
“Specifically, we follow prior research that leverages the probability score of a classifier as a confidence score (e.g. probabilities from the softmax layer of a neural network) [23, 46, 61] and utilize the predicted score of a model, P M ( Y | X ) to identify low confidence samples (e.g. 0 .”
MethodPower to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084 · cited 1×
“Human-in-the-loop, interactive ML approaches have been actively explored by researchers to create a better ML model with improved performance and user acceptance [1, 12, 37, 42] and improve the annotations [28, 32, 35].”
BackgroundUnintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066 · cited 1×
“However, the development and integration of these systems in practice remain a challenge due to the lack of human-centered designs and performing as a "black-box" system [11–13, 17, 21, 31, 40, 60].”
Method
Research works printed but not linked: 9
- Mark T Bayley , Amanda Hurdowar , Carol L Richards , Nicol Korner-Bitensky , Sharon Wood-Dauphinee , Janice J Eng , Marilyn McKay-Lyons, Edward Harrison , Robert Teasell, Margaret Harrison , 2012 . Barriers to implementation of stroke rehabilitation evidence: findings from a multi-site pilot project. Disability and rehabilitation 34, 19 (2012), 1633–1638. Mark T Bayley, Amanda Hurdowar, Carol L Richards, Nicol Korner-Bitensky, Sharon Wood-Dauphinee, Janice J Eng, Marilyn McKay-Lyons, Edward Harrison, Robert Teasell, Margaret Harrison, 2012. Barriers to implementation of stroke rehabilitation evidence: findings from a multi-site pilot project. Disability and rehabilitation 34, 19 (2012), 1633–1638.
- Or Biran and Courtenay Cotton . 2017 . Explanation and justification in machine learning: A survey . In IJCAI-17 workshop on explainable AI (XAI), Vol. 8. 1. Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In IJCAI-17 workshop on explainable AI (XAI), Vol. 8. 1.
- Po- Hsuan Cameron Chen , Yun Liu , and Lily Peng . 2019. How to develop machine learning models for healthcare. Nature materials 18, 5 ( 2019 ), 410. Po-Hsuan Cameron Chen, Yun Liu, and Lily Peng. 2019. How to develop machine learning models for healthcare. Nature materials 18, 5 (2019), 410.
- Andre Esteva , Brett Kuprel , Roberto A Novoa , Justin Ko , Susan M Swetter , Helen M Blau , and Sebastian Thrun . 2017. Dermatologist-level classification of skin cancer with deep neural networks. nature 542, 7639 ( 2017 ), 115–118. Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. 2017. Dermatologist-level classification of skin cancer with deep neural networks. nature 542, 7639 (2017), 115–118.
- Chuan Guo , Geoff Pleiss , Yu Sun , and Kilian Q Weinberger . 2017 . On calibration of modern neural networks . In International Conference on Machine Learning. PMLR, 1321–1330 . Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International Conference on Machine Learning. PMLR, 1321–1330.
- Saif Khairat , David Marc , William Crosby , and Ali Al Sanousi . 2018. Reasons for physicians not adopting clinical decision support systems: critical analysis. JMIR medical informatics 6, 2 ( 2018 ), e24. Saif Khairat, David Marc, William Crosby, and Ali Al Sanousi. 2018. Reasons for physicians not adopting clinical decision support systems: critical analysis. JMIR medical informatics 6, 2 (2018), e24.
- Been Kim Julie A Shah and Finale Doshi-Velez. 2015. Mind the gap: A generative approach to interpretable feature selection and extraction. In Advances in Neural Information Processing Systems. 2260–2268. Been Kim Julie A Shah and Finale Doshi-Velez. 2015. Mind the gap: A generative approach to interpretable feature selection and extraction. In Advances in Neural Information Processing Systems. 2260–2268.
- Jon Kleinberg , Himabindu Lakkaraju , Jure Leskovec , Jens Ludwig , and Sendhil Mullainathan . 2018. Human decisions and machine predictions. The quarterly journal of economics 133, 1 ( 2018 ), 237–293. Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 2018. Human decisions and machine predictions. The quarterly journal of economics 133, 1 (2018), 237–293.
- Edward Taub , David M Morris , Jean Crago , Danna Kay King , Mary Bowman , Camille Bryson , Staci Bishop , Sonya Pearson , and Sharon E Shaw . 2011. Wolf motor function test (WMFT) manual. Birmingham: University of Alabama , CI Therapy Research Group ( 2011 ). Edward Taub, David M Morris, Jean Crago, Danna Kay King, Mary Bowman, Camille Bryson, Staci Bishop, Sonya Pearson, and Sharon E Shaw. 2011. Wolf motor function test (WMFT) manual. Birmingham: University of Alabama, CI Therapy Research Group (2011).
Cited by
What is Human-Centered about Human-Centered AI? A Map of the Research Landscape
Capel, T., & Brereton, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580959 · Cited by 228
“…these human-AI collaborative teams in various capacities and domains [3, 14, 23, 24, 36, 49, 58, 61, 64, 66, 76, 82, 100, 116, 117, 128, 139, 143, 149, 163, 164, 179, 181, 184, 189, 201, 205, 217, 218, 220, 223, 254, 255, 260, 266–270, 275, 276, 287, 289, 293, 299, 302, 308, 317–319, 322].”
BackgroundInfluentialUnderstanding Human-Centred AI: a review of its defining elements and a research agenda
Schmager, S., Pappas, I. O., & Vassilakopoulou, P. (2025) · Behaviour and Information Technology · doi:10.1080/0144929x.2024.2448719 · Cited by 254
Lee, M. H., & Chew, C. J. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610218 · Cited by 78
“Specifically, researchers have discussed the potential of human and AI/ML teaming to achieve better results than either humans or AI/ML models alone [5, 42, 43].”
BackgroundSelf-citationUymaz, P., Uymaz, A. O., & Akgül, Y. (2023) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2023.2233126 · Cited by 24
“ANN approaches issues via the lens of inputs, outputs, and the weights of variables that link inputs and outputs (Davenport & Kalakota, 2019; Lee et al., 2022).”
Measuring User Experience Inclusivity in Human-AI Interaction via Five User Problem-Solving Styles
Anderson, A. W., Guevara, J. N., Moussaoui, F., Li, T., Vorvoreanu, M., & Burnett, M. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3663740 · Cited by 38
Metadata
- Authors
- Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S.
- Affiliations
- Singapore Management University (SG)
- Carnegie Mellon University (US)
- University of Lisbon (PT)
- Faculdade de Ciências Exatas e da Engenharia, Universidade da Madeira, Portugal
- Venue
- 27th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2022-03-22
- Pages
- 4-14
- OpenAlex
- W4220855893
- Cited by
- 19
- References
- 48
- Citations by year
- 2026: 5 · 2025: 6 · 2024: 4 · 2023: 4
- FWCI
- 12.48
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion, cc-by-nc-nd)
- Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) (publishedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, other-oa)
- Funders
- LISBOA 2020 and the FCT (LISBOA-01-0247-FEDER-045948)
- Singapore Ministry of Education (MOE) (21-SIS-SMU-034)
- National Science Foundation (CNS-1518865)
- FCT (UIDB/50009/2020)
- Topics
- Health Sciences › Medicine › Rehabilitation › Stroke Rehabilitation and Recovery
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- clinical decision support systems, human-AI collaboration, stroke rehabilitation assessment, physiotherapy, rule-based models, domain expert knowledge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Towards Efficient Annotations for a Human-AI Collaborative, Clinical Decision Support System: A Case Study on Physical Stroke Rehabilitation Assessment. https://app.cochat.ai/papers/10.1145/3490099.3511112
- The paper
- Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. (2022). Towards Efficient Annotations for a Human-AI Collaborative, Clinical Decision Support System: A Case Study on Physical Stroke Rehabilitation Assessment. 27th International Conference on Intelligent User Interfaces. https://doi.org/10.1145/3490099.3511112
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.