Towards Efficient Annotations for a Human-AI Collaborative, Clinical Decision Support System: A Case Study on Physical Stroke Rehabilitation Assessment
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
Artificial intelligence (AI) and machine learning (ML) algorithms are increasingly being explored to support various decision-making tasks in health (e.g. rehabilitation assessment). However, the development of such AI/ML-based decision support systems is challenging due to the expensive process to collect an annotated dataset. In this paper, we describe the development process of a human-AI collaborative, clinical decision support system that augments an ML model with a rule-based (RB) model from domain experts. We conducted its empirical evaluation in the context of assessing physical stroke rehabilitation with the dataset of three exercises from 15 post-stroke survivors and therapists. Our results bring new insights on the efficient development and annotations of a decision support system: when an annotated dataset is not available initially, the RB model can be used to assess post-stroke survivor’s quality of motion and identify samples with low confidence scores to support efficient annotations for training an ML model. Specifically, our system requires only 22 - 33% of annotations from therapists to train an ML model that achieves equally good performance with an ML model with all annotations from a therapist. Our work discusses the values of a human-AI collaborative approach for effectively collecting an annotated dataset and supporting a complex decision-making task.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“researchers have explored the feasibility of human-AI collaborative decision-making, in which an AI model provides humans new data-driven insights on a task for achieving complementary performance, outperforming neither of the AI or the human alone [5, 30, 42, 43].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [43]
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · cited 2×
“To address this latter issue, researchers have actively explored a method or tool to provide explanations on the predictions of anML-based system [25, 40, 43, 50].”
MethodRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“Although ML-based systems have shown competitive performance, prior work describes the difficulty to integrate these systems into practice due to the lack of human-centered designs as a primary reason for failure [13, 31, 40, 60] and performing as a black-box system [52].”
BackgroundMultimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · cited 2×
“Thus, it would be interesting to further explore whether the HM can support training a better ML model and achieve more efficient annotation after improving the technique of a multimodal machine learning [4] and fine-tuning the RB model.”
BackgroundMethodLanghorne, P., Bernhardt, J., & Kwakkel, G. (2011) · The Lancet · doi:10.1016/s0140-6736(11)60325-5 · Cited by 2,726
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“When this interface presents the predicted assessment, it also includes the confidence score to “make clear how well the system can do” [2].”
BackgroundPredicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884 · cited 1×
“probabilities from the softmax layer of a neural network) [23, 46, 61] and utilize the predicted score of a model, PM (Y |X ) to identify low confidence samples (e.”
MethodIntelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · cited 2×
“Recent advances in artificial intelligence (AI) have made it applicable to support decision-making in healthcare [6, 12, 14, 42].”
BackgroundTransforming classifier scores into accurate multiclass probability estimates
Zadrozny, B., & Elkan, C. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775151 · Cited by 1,280 · cited 1×
“Specifically, we follow prior research that leverages the probability score of a classifier as a confidence score (e.g. probabilities from the softmax layer of a neural network) [23, 46, 61] and utilize the predicted score of a model, P M ( Y | X ) to identify low confidence samples (e.g. 0 .”
MethodPower to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084 · cited 1×
“Human-in-the-loop, interactive ML approaches have been actively explored by researchers to create a better ML model with improved performance and user acceptance [1, 12, 37, 42] and improve the annotations [28, 32, 35].”
BackgroundUnintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066 · cited 1×
“However, the development and integration of these systems in practice remain a challenge due to the lack of human-centered designs and performing as a "black-box" system [11–13, 17, 21, 31, 40, 60].”
MethodMovement Smoothness Changes during Stroke Recovery
Rohrer, B., Fasoli, S., Krebs, H. I., Hughes, R., Volpe, B., Frontera, W. R., Stein, J., & Hogan, N. (2002) · The Journal of Neuroscience · doi:10.1523/jneurosci.22-18-08297.2002 · Cited by 767 · cited 2×
“acceleration and jerk, and Mean Arrest Period Ratio (the portion of the frames when speed exceeds 10% of the maximum speed) [51].”
BackgroundMethodLetham, B., Rudin, C., McCormick, T. H., & Madigan, D. (2015) · The Annals of Applied Statistics · doi:10.1214/15-aoas848 · Cited by 766 · cited 3×
“…decision support systems in high-stake domains: at the beginning of developing these systems, we recommend researchers engaging with domain experts to develop an interpretable, rule-based (RB) model [41, 43, 49] and utilize it to support efficient annotations for training an ML model.”
BackgroundMethodA Review of Computer Vision Techniques for the Analysis of Urban Traffic
Buch, N., Velastin, S. A., & Orwell, J. (2011) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2011.2119372 · Cited by 653 · cited 1×
“An additional study is required to expand the application of our human-AI collaborative approach on different tasks or data modalities [9, 19].”
BackgroundJAABA: Interactive machine learning for automatic annotation of animal behavior
Kabra, M., Robie, A. A., Rivera-Alba, M., Branson, S., & Branson, K. (2013) · Nature Methods · doi:10.1038/nmeth.2281 · Cited by 644
Beede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., & Vardoulakis, L. M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376718 · Cited by 614 · cited 6×
“In addition, there is increasing recent research efforts [6, 13, 21, 40, 54, 60] that highlight the importance of involving stakeholders to understand their practices and needs [13, 40, 60] and socio-environmental factors [6] for the design and evaluation of a system.”
BackgroundInfluentialThe need for uncertainty quantification in machine-assisted medical decision making
Begoli, E., Bhattacharya, T., & Kusnezov, D. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-018-0004-1 · Cited by 597 · cited 1×
“…with low confidence scores [23], it is important to further investigate an approach to estimate the confidence score of a model and determine if the prediction of a model can be utilized it to better identify samples with low confidence scores and build a trustful interaction with the user [7, 23].”
BackgroundCai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · cited 6×
“In addition, there is increasing recent research efforts [6, 13, 21, 40, 54, 60] that highlight the importance of involving stakeholders to understand their practices and needs [13, 40, 60] and socio-environmental factors [6] for the design and evaluation of a system.”
BackgroundMethodInfluentialVisual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers
Hohman, F., Kahng, M., Pienta, R., & Chau, D. H. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2843369 · Cited by 582 · cited 2×
“To address this latter issue, researchers have actively explored a method or tool to provide explanations on the predictions of anML-based system [25, 40, 43, 50].”
MethodPrinciples of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576 · cited 3×
“Prior work shows the feasibility of presenting relevant information of a task and acquiring inputs of a user (e.g. constraints of a model [29] or feature relevance [37, 42]) to refine an ML model.”
BackgroundPeiffer-Smadja, N., Rawson, T. M., Ahmad, R., Buchard, A., Georgiou, P., Lescure, F.-X., Birgand, G., & Holmes, A. H. (2020) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2019.09.009 · Cited by 561 · cited 2×
“However, it remains challenging to elicit a global set of rules that represent the knowledge of clinicians on a complex decision-making task [41, 49].”
BackgroundMethodSullivan, K. J., Tilson, J. K., Cen, S. Y., Rose, D. K., Hershberg, J., Correa, A., Gallichio, J., McLeod, M., Moore, C., Wu, S. S., et al. (2011) · Stroke · doi:10.1161/strokeaha.110.592766 · Cited by 445 · cited 7×
“Fugl Meyer Assessment [57]), we utilized the annotations of TP 1 as ground truth for the development and evaluation of a system.”
BackgroundMethodInfluentialSystematic review of Kinect applications in elderly care and stroke rehabilitation
Webster, D., & Celik, O. (2014) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/1743-0003-11-108 · Cited by 436 · cited 1×
“Researchers have explored an approach of automatically monitoring and assessing patient’s exercises with machine learning and sensors to improve current practices of rehabilitation [59].”
BackgroundHuman-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405 · cited 5×
“tively explored by researchers to create a better ML model with improved performance and user acceptance [1, 12, 37, 42] and improve the annotations [28, 32, 35].”
BackgroundInfluentialThe Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · cited 3×
“It is necessary to further study how and what additional interactions can be provided to enable trustful usage by the domain experts and improve their decision making without introducing biases [21].”
BackgroundClinical decision support system: Risk level prediction of heart disease using weighted fuzzy rules
Anooj, P. K. (2012) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2011.09.002 · Cited by 385 · cited 1×
“Clinical decision support systems can be categorized as either a rule-based system [3, 10, 41, 55] or a machine learning-based system [6, 12, 22, 39, 53], depending on its data analytic techniques.”
BackgroundBias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287572 · Cited by 337 · cited 1×
“Machine learning (ML) algorithms are increasingly being explored to support human decision-making in high-stakes situations, such as public services, criminal justice, and health [6, 12, 14, 18, 30, 36].”
BackgroundBias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · arXiv · doi:10.48550/arxiv.1901.09451 · Cited by 337
Yang, Q., Steinfeld, A., & Zimmerman, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300468 · Cited by 247 · cited 4×
“conducted a field evaluation on the design of a clinical decision support tool for cardiologists with synthetic data and found that clinicians are more likely to embrace a tool that augments their decision-making in natural and intuitive ways [60].”
BackgroundInfluentialConfidence-based active learning
Li, M., & Sethi, I. K. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.156 · Cited by 230
A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
De-Arteaga, M., Fogliato, R., & Chouldechova, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376638 · Cited by 221 · cited 3×
“Following prior research that describes the potential of ML algorithms to outperform human experts on prediction tasks [17, 20, 34, 39], researchers have investigated the feasibility of deploying these ML-based decision support systems to improve decision makings.”
BackgroundMethod"The human body is a black box": supporting clinical decision-making with deep learning
Sendak, M., Elish, M. C., Gao, M., Futoma, J., Ratliff, W., Nichols, M., Bedoya, A., Balu, S., & O'Brien, C. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372827 · Cited by 196 · cited 2×
“In addition, researchers discuss the necessity of evaluating a system in socio-technical contexts [6, 21], and creating ongoing feedback loops with stakeholders [54].”
BackgroundGultepe, E., Green, J. P., Nguyen, H., Adams, J., Albertson, T., & Tagkopoulos, I. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-001815 · Cited by 187 · cited 1×
“Clinical decision support systems can be categorized as either a rule-based system [3, 10, 41, 55] or a machine learning-based system [6, 12, 22, 39, 53], depending on its data analytic techniques.”
BackgroundMachine learning based decision support systems (DSS) for heart disease diagnosis: a review
Safdar, S., Zafar, S. N., Zafar, N., & Khan, N. F. (2017) · Artificial Intelligence Review · doi:10.1007/s10462-017-9552-8 · Cited by 183 · cited 1×
“Clinical decision support systems can be categorized as either a rule-based system [3, 10, 41, 55] or a machine learning-based system [6, 12, 22, 39, 53], depending on its data analytic techniques.”
BackgroundPanwar, M., Biswas, D., Bajaj, H., Jobges, M., Turk, R., Maharatna, K., & Acharyya, A. (2019) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2019.2899927 · Cited by 158 · cited 3×
“An alternative approach is to utilize machine learning algorithms with labeled sensor data and learn a model to assess the quality of motion [39, 48].”
BackgroundMethodInteractive optimization for steering machine classification
Kapoor, A., Lee, B., Tan, D. S., & Horvitz, E. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753529 · Cited by 138 · cited 1×
“Prior work shows the feasibility of presenting relevant information of a task and acquiring inputs of a user (e.g. constraints of a model [29] or feature relevance [37, 42]) to refine an ML model.”
BackgroundA Human-AI Collaborative Approach for Clinical Decision Making on Rehabilitation Assessment
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445472 · Cited by 137
Self-citationUnderstanding and Visualizing Data Iteration in Machine Learning
Hohman, F., Wongsuphasawat, K., Kery, M. B., & Patel, K. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376177 · Cited by 110 · cited 1×
“In addition, researchers have discussed the importance of iterating on data (e.g. collecting new data or labels) [26] and explored interactive approaches for annotations in various applications: sound event detection [32], semantic annotations [35], and animal behavior [28].”
BackgroundDeveloping healthcare rule-based expert systems: Case study of a heart failure telemonitoring system
Seto, E., Leonard, K. J., Cafazzo, J. A., Barnsley, J., Masino, C., & Ross, H. J. (2012) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2012.03.001 · Cited by 104 · cited 2×
“Clinical decision support systems can be categorized as either a rule-based system [3, 10, 41, 55] or a machine learning-based system [6, 12, 22, 39, 53], depending on its data analytic techniques.”
BackgroundDas, S., Trutoiu, L., Murai, A., Alcindor, D., Oh, M., De la Torre, F., & Hodgins, J. (2011) · 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society · doi:10.1109/iembs.2011.6091674 · Cited by 84 · cited 1×
“applied Support Vector Machine (SVM) to distinguish mild and severe symptoms of Parkinson’s Disease using full-body motion capture data from four Parkinson’s patients [16].”
MethodRehabilitation practice: challenges to effective team working
Long, A., Kneafsey, R., & Ryan, J. (2003) · International Journal of Nursing Studies · doi:10.1016/s0020-7489(03)00015-4 · Cited by 79 · cited 1×
“Although rehabilitation assessment is important to determine a customized intervention of a patient, it is infrequently performed due to the therapist’s limited availability [45].”
BackgroundIntelligent decision-making support systems : foundations, applications and challenges
Gupta, J. N. D., Forgionne, G. A., & Mora, M. (2006) · doi:10.1007/1-84628-231-4 · Cited by 73
Learning to assess the quality of stroke rehabilitation exercises
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Badia, S. B. I. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302273 · Cited by 71 · cited 10×
“This work leverages a Kinect v2 sensor (Microsoft, Redmond, USA) to track the joint positions of a post-stroke survivor while performing an exercise and extracts various kinematic features to represent the quality of motion [39, 51, 57, 58].”
BackgroundMethodInfluentialSelf-citationChallenges in Applying Best Evidence to Physiotherapy
Jones, M. A., Grimmer, K. A., Edwards, I., Higgs, J., & Trede, F. (2006) · Internet Journal of Allied Health Sciences and Practice · doi:10.46743/1540-580x/2006.1117 · Cited by 70
Allocating Interventions Based on Predicted Outcomes: A Case Study on Homelessness Services
Kube, A., Das, S., & Fowler, P. J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301622 · Cited by 62 · cited 1×
“Machine learning (ML) algorithms are increasingly being explored to support human decision-making in high-stakes situations, such as public services, criminal justice, and health [6, 12, 14, 18, 30, 36].”
BackgroundA Human-in-the-Loop System for Sound Event Detection and Annotation
Kim, B., & Pardo, B. (2018) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3214366 · Cited by 60 · cited 2×
“In addition, researchers have discussed the importance of iterating on data (e.g. collecting new data or labels) [26] and explored interactive approaches for annotations in various applications: sound event detection [32], semantic annotations [35], and animal behavior [28].”
BackgroundLee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415227 · Cited by 57
Self-citationKurosu, M., & Hashizume, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35596-7 · Cited by 27
An Exploratory Study on Techniques for Quantitative Assessment of Stroke Rehabilitation Exercises
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A. J. M., & Badia, S. B. I. (2020) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3340631.3394872 · Cited by 25 · cited 8×
“However, it remains challenging to elicit a global set of rules that represent the knowledge of clinicians on a complex decision-making task [41, 49].”
BackgroundMethodInfluentialSelf-citationMcgee, W. C. (1963) · Proceedings of the IEEE · doi:10.1109/proc.1963.2243 · Cited by 17
Dhivyaprabha, T. T., Subashini, P., & Krishnaveni, M. (2016) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci.2016.7850050 · Cited by 17 · cited 1×
“An additional study is required to expand the application of our human-AI collaborative approach on different tasks or data modalities [9, 19].”
Background
Research works printed but not linked: 9
- Mark T Bayley , Amanda Hurdowar , Carol L Richards , Nicol Korner-Bitensky , Sharon Wood-Dauphinee , Janice J Eng , Marilyn McKay-Lyons, Edward Harrison , Robert Teasell, Margaret Harrison , 2012 . Barriers to implementation of stroke rehabilitation evidence: findings from a multi-site pilot project. Disability and rehabilitation 34, 19 (2012), 1633–1638. Mark T Bayley, Amanda Hurdowar, Carol L Richards, Nicol Korner-Bitensky, Sharon Wood-Dauphinee, Janice J Eng, Marilyn McKay-Lyons, Edward Harrison, Robert Teasell, Margaret Harrison, 2012. Barriers to implementation of stroke rehabilitation evidence: findings from a multi-site pilot project. Disability and rehabilitation 34, 19 (2012), 1633–1638.
- Or Biran and Courtenay Cotton . 2017 . Explanation and justification in machine learning: A survey . In IJCAI-17 workshop on explainable AI (XAI), Vol. 8. 1. Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In IJCAI-17 workshop on explainable AI (XAI), Vol. 8. 1.
- Po- Hsuan Cameron Chen , Yun Liu , and Lily Peng . 2019. How to develop machine learning models for healthcare. Nature materials 18, 5 ( 2019 ), 410. Po-Hsuan Cameron Chen, Yun Liu, and Lily Peng. 2019. How to develop machine learning models for healthcare. Nature materials 18, 5 (2019), 410.
- Andre Esteva , Brett Kuprel , Roberto A Novoa , Justin Ko , Susan M Swetter , Helen M Blau , and Sebastian Thrun . 2017. Dermatologist-level classification of skin cancer with deep neural networks. nature 542, 7639 ( 2017 ), 115–118. Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. 2017. Dermatologist-level classification of skin cancer with deep neural networks. nature 542, 7639 (2017), 115–118.
- Chuan Guo , Geoff Pleiss , Yu Sun , and Kilian Q Weinberger . 2017 . On calibration of modern neural networks . In International Conference on Machine Learning. PMLR, 1321–1330 . Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International Conference on Machine Learning. PMLR, 1321–1330.
- Saif Khairat , David Marc , William Crosby , and Ali Al Sanousi . 2018. Reasons for physicians not adopting clinical decision support systems: critical analysis. JMIR medical informatics 6, 2 ( 2018 ), e24. Saif Khairat, David Marc, William Crosby, and Ali Al Sanousi. 2018. Reasons for physicians not adopting clinical decision support systems: critical analysis. JMIR medical informatics 6, 2 (2018), e24.
- Been Kim Julie A Shah and Finale Doshi-Velez. 2015. Mind the gap: A generative approach to interpretable feature selection and extraction. In Advances in Neural Information Processing Systems. 2260–2268. Been Kim Julie A Shah and Finale Doshi-Velez. 2015. Mind the gap: A generative approach to interpretable feature selection and extraction. In Advances in Neural Information Processing Systems. 2260–2268.
- Jon Kleinberg , Himabindu Lakkaraju , Jure Leskovec , Jens Ludwig , and Sendhil Mullainathan . 2018. Human decisions and machine predictions. The quarterly journal of economics 133, 1 ( 2018 ), 237–293. Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 2018. Human decisions and machine predictions. The quarterly journal of economics 133, 1 (2018), 237–293.
- Edward Taub , David M Morris , Jean Crago , Danna Kay King , Mary Bowman , Camille Bryson , Staci Bishop , Sonya Pearson , and Sharon E Shaw . 2011. Wolf motor function test (WMFT) manual. Birmingham: University of Alabama , CI Therapy Research Group ( 2011 ). Edward Taub, David M Morris, Jean Crago, Danna Kay King, Mary Bowman, Camille Bryson, Staci Bishop, Sonya Pearson, and Sharon E Shaw. 2011. Wolf motor function test (WMFT) manual. Birmingham: University of Alabama, CI Therapy Research Group (2011).
Cited by
What is Human-Centered about Human-Centered AI? A Map of the Research Landscape
Capel, T., & Brereton, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580959 · Cited by 228
“…these human-AI collaborative teams in various capacities and domains [3, 14, 23, 24, 36, 49, 58, 61, 64, 66, 76, 82, 100, 116, 117, 128, 139, 143, 149, 163, 164, 179, 181, 184, 189, 201, 205, 217, 218, 220, 223, 254, 255, 260, 266–270, 275, 276, 287, 289, 293, 299, 302, 308, 317–319, 322].”
BackgroundInfluentialUnderstanding Human-Centred AI: a review of its defining elements and a research agenda
Schmager, S., Pappas, I. O., & Vassilakopoulou, P. (2025) · Behaviour and Information Technology · doi:10.1080/0144929x.2024.2448719 · Cited by 254
Lee, M. H., & Chew, C. J. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610218 · Cited by 78
“Specifically, researchers have discussed the potential of human and AI/ML teaming to achieve better results than either humans or AI/ML models alone [5, 42, 43].”
BackgroundSelf-citationUymaz, P., Uymaz, A. O., & Akgül, Y. (2023) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2023.2233126 · Cited by 24
“ANN approaches issues via the lens of inputs, outputs, and the weights of variables that link inputs and outputs (Davenport & Kalakota, 2019; Lee et al., 2022).”
Measuring User Experience Inclusivity in Human-AI Interaction via Five User Problem-Solving Styles
Anderson, A. W., Guevara, J. N., Moussaoui, F., Li, T., Vorvoreanu, M., & Burnett, M. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3663740 · Cited by 38
Metadata
- Authors
- Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S.
- Affiliations
- Singapore Management University (SG)
- Carnegie Mellon University (US)
- University of Lisbon (PT)
- Faculdade de Ciências Exatas e da Engenharia, Universidade da Madeira, Portugal
- Venue
- 27th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2022-03-22
- Pages
- 4-14
- OpenAlex
- W4220855893
- Cited by
- 19
- References
- 48
- Citations by year
- 2026: 5 · 2025: 6 · 2024: 4 · 2023: 4
- FWCI
- 12.48
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion, cc-by-nc-nd)
- Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) (publishedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, other-oa)
- Funders
- LISBOA 2020 and the FCT (LISBOA-01-0247-FEDER-045948)
- Singapore Ministry of Education (MOE) (21-SIS-SMU-034)
- National Science Foundation (CNS-1518865)
- FCT (UIDB/50009/2020)
- Topics
- Health Sciences › Medicine › Rehabilitation › Stroke Rehabilitation and Recovery
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- clinical decision support systems, human-AI collaboration, stroke rehabilitation assessment, physiotherapy, rule-based models, domain expert knowledge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.