Co-Design and Evaluation of an Intelligent Decision Support System for Stroke Rehabilitation Assessment
Open accessAbstract
Clinical decision support systems have the potential to improve work flows of experts in practice (e.g. therapist's evidence-based rehabilitation assessment). However, the adoption of these systems is challenging, and the gains of these systems have not fully demonstrated yet. In this paper, we identified the needs of therapists to assess patient's functional abilities (e.g. alternative perspectives with quantitative information on patient's exercise motions). As a result, we co-designed and developed an intelligent decision support system that automatically identifies salient features of assessment using reinforcement learning to assess the quality of motion and generate patient-specific analysis. We evaluated this system with seven therapists using the dataset from 15 patients performing three exercises. The results show that therapists have higher usage intent on our system than a traditional system without patient-specific analysis ($p < 0.05$). While presenting richer information ($p < 0.10$), our system significantly reduces therapists' effort on assessment ($p < 0.10$) and improves their agreement on assessment from 0.66 to 0.71 F1-scores ($p < 0.01$). This work discusses the importance of human centered design and development of a machine learning-based decision support system that presents contextually relevant information and salient explanations on its prediction for better adoption in practice.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Therapists preferred to review feature-based explanations on rehabilitation assessment tasks.
“First, the previous research describes that therapists preferred to review feature-based explanations on rehabilitation assessment tasks [41].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [41]
Involving stakeholders is important for the design and evaluation of a system.
“there has been increasing recent research efforts [8, 15, 22, 41, 58, 65] that highlight the importance of involving stakeholders to understand their practices and needs [15, 41, 65] and socio-environmental factors [8] for the design and evaluation of a system.”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [41]
Only the top three salient features with the highest scores were utilized to avoid overwhelming users.
“we identified patient-specific, salient features and utilized only the top three salient features with the highest scores to avoid overwhelming users [35, 41].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [41]
Therapists preferred to review feature-based explanations to find evidence and confirm their assessment.
“As previous work describes therapists preferred to review feature-based explanations to find evidence and confirm their assessment [41]”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [41]
Such a small sample size is not unusual in similar previous works.
“However, such a small sample size is not unusual in similar previous works [11, 41].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [41]
Several mechanisms have been proposed to optimize the combination of human and AI decisions and facilitate appropriate reliance, including interpretable explanations, cognitive forcing functions, and feedback loops.
“These mechanisms include providing interpretable explanations for AI advice [21, 72, 123], using cognitive forcing functions[17, 47, 99], and incorporating feedback loops to enhance the interaction between humans and AI systems [10, 11, 139].”
From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [72]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877 · see citing works
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003 · see citing works
(2017) · Cited by 11,625 · see citing works
Gale, N. K., Heath, G., Cameron, E., Rashid, S., & Redwood, S. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-117 · Cited by 10,865 · see citing works
Gene selection for cancer classification using support vector machines
Guyon, I., Weston, J., Barnhill, S., & Vapnik, V. (2002) · Machine Learning · doi:10.1023/a:1012487302797 · Cited by 9,954 · see citing works
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539 · see citing works
Parkinson's disease: clinical features and diagnosis
Jankovic, J. (2008) · Journal of Neurology, Neurosurgery & Psychiatry · doi:10.1136/jnnp.2007.131045 · Cited by 5,720 · see citing works
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318 · see citing works
Garg, A. X., Adhikari, N. K. J., McDonald, H., Rosas-Arellano, M. P., Devereaux, P. J., Beyene, J., Sam, J., & Haynes, R. B. (2005) · JAMA · doi:10.1001/jama.293.10.1223 · Cited by 2,936 · see citing works
Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust
Hoff, K. A., & Bashir, M. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720814547570 · Cited by 2,817 · see citing works
Kawamoto, K., Houlihan, C. A., Balas, E. A., & Lobach, D. F. (2005) · BMJ · doi:10.1136/bmj.38398.500764.8f · Cited by 2,703 · see citing works
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · see citing works
Feigin, V. L., Norrving, B., & Mensah, G. A. (2017) · Circulation Research · doi:10.1161/circresaha.116.308413 · Cited by 2,184 · see citing works
Unintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066 · see citing works
Human motion tracking for rehabilitation-A survey
Zhou, H., & Hu, H. (2008) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2007.09.001 · Cited by 813 · see citing works
Movement Smoothness Changes during Stroke Recovery
Rohrer, B., Fasoli, S. E., Krebs, H. I., Hughes, R. L., Volpe, B. T., Frontera, W. R., Stein, J. M., & Hogan, N. (2002) · Journal of Neuroscience · doi:10.1523/jneurosci.22-18-08297.2002 · Cited by 767 · see citing works
Fails, J. A., & Olsen, D. R. (2003) · Proceedings of the 8th international conference on Intelligent user interfaces · doi:10.1145/604045.604056 · Cited by 633 · see citing works
How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface
Kizilcec, R. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858402 · Cited by 615 · see citing works
Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · see citing works
Principles of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576 · see citing works
Exoskeleton robots for upper-limb rehabilitation: State of the art and future prospects
Lo, H. S., & Xie, S. Q. (2012) · Medical Engineering & Physics · doi:10.1016/j.medengphy.2011.10.004 · Cited by 571 · see citing works
Krause, J., Gulshan, V., Rahimy, E., Karth, P., Widner, K., Corrado, G. S., Peng, L., & Webster, D. R. (2018) · Ophthalmology · doi:10.1016/j.ophtha.2018.01.034 · Cited by 564 · see citing works
Treatment Techniques and Clinical Guidelines for Photocoagulation of Diabetic Macular Edema
(1987) · Ophthalmology · doi:10.1016/s0161-6420(87)33527-4 · Cited by 544 · see citing works
The clinical utility of posturography
Visser, J. E., Carpenter, M. G., van der Kooij, H., & Bloem, B. R. (2008) · Clinical Neurophysiology · doi:10.1016/j.clinph.2008.07.220 · Cited by 490 · see citing works
Clinical applications of machine learning algorithms: beyond the black box
Watson, D. S., Krutzinna, J., Bruce, I. N., Griffiths, C. E., McInnes, I. B., Barnes, M. R., & Floridi, L. (2019) · BMJ · doi:10.1136/bmj.l886 · Cited by 450 · see citing works
Sullivan, K. J., Tilson, J. K., Cen, S. Y., Rose, D. K., Hershberg, J., Correa, A., Gallichio, J., McLeod, M., Moore, C., Wu, S. S., et al. (2011) · Stroke · doi:10.1161/strokeaha.110.592766 · Cited by 445 · see citing works
Systematic review of Kinect applications in elderly care and stroke rehabilitation
Webster, D., & Celik, O. (2014) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/1743-0003-11-108 · Cited by 436 · see citing works
A Socially Assistive Robot Exercise Coach for the Elderly
Fasola, J., & Matarić, M. J. (2013) · Journal of Human-Robot Interaction · doi:10.5898/jhri.2.2.fasola · Cited by 422 · see citing works
Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405 · see citing works
Clinical decision support system: Risk level prediction of heart disease using weighted fuzzy rules
Anooj, P. (2011) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2011.09.002 · Cited by 384 · see citing works
Baig, M. M., GholamHosseini, H., Moqeem, A. A., Mirza, F., & Lindén, M. (2017) · Journal of Medical Systems · doi:10.1007/s10916-017-0760-1 · Cited by 351 · see citing works
Reasons For Physicians Not Adopting Clinical Decision Support Systems: Critical Analysis
Khairat, S., Marc, D. T., Crosby, W., & Sanousi, A. A. (2018) · JMIR Medical Informatics · doi:10.2196/medinform.8912 · Cited by 336 · see citing works
The evolution of clinical gait analysis
Sutherland, D. H. (2002) · Gait & Posture · doi:10.1016/s0966-6362(02)00004-8 · Cited by 315 · see citing works
Reviewing intuitive decision‐making and uncertainty: the implications for medical education
Hall, K. (2002) · Medical Education · doi:10.1046/j.1365-2923.2002.01140.x · Cited by 301 · see citing works
Towards Pervasive Physical Rehabilitation Using Microsoft Kinect
Chang, C., Lange, B. S., Zhang, M., Koenig, S. T., Requejo, P., Somboon, N., Sawchuk, A. A., & Rizzo, A. (2012) · International Conference on Pervasive Computing Technologies for Healthcare · doi:10.4108/icst.pervasivehealth.2012.248714 · Cited by 263 · see citing works
Task‐oriented training in rehabilitation after stroke: systematic review
Rensink, M., Schuurmans, M. J., Lindeman, E., & Hafsteinsdøttir, T. B. (2009) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.2008.04925.x · Cited by 262 · see citing works
Yang, Q., Steinfeld, A., & Zimmerman, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300468 · Cited by 247 · see citing works
Virtual reality using games for improving physical functioning in older adults: a systematic review
Molina, K. I., Ricci, N. A., de Moraes, S. A., & Perracini, M. R. (2014) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/1743-0003-11-156 · Cited by 245 · see citing works
Radar plots: a useful way for presenting multivariate health care data
Saary, M. J. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.04.021 · Cited by 227 · see citing works
Explainable Artificial Intelligence (XAI)
Mohaghegh, S. D. (2024) · CRC Press eBooks · doi:10.1201/9781003369356-8 · Cited by 222 · see citing works
Clinical Decision Support Systems
(2016) · Health informatics · doi:10.1007/978-3-319-31913-1 · Cited by 196 · see citing works
Wu, C., Trombly, C. A., Lin, K., & Tickle‐Degnen, L. (2000) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/s0003-9993(00)90228-4 · Cited by 189 · see citing works
Panwar, M., Biswas, D., Bajaj, H., Jobges, M., Turk, R., Maharatna, K., & Acharyya, A. (2019) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2019.2899927 · Cited by 158 · see citing works
Barnett, M. L., Boddupalli, D., Nundy, S., & Bates, D. W. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.0096 · Cited by 154 · cited 2× · see citing works
“As experts often rely on their experience-based heuristics, disagreement on decision making is prevalent [4, 30, 31].”
BackgroundKinect-enabled home-based rehabilitation system using Dynamic Time Warping and fuzzy logic
Su, C.-J., Chiang, C.-Y., & Huang, J.-Y. (2014) · Applied Soft Computing · doi:10.1016/j.asoc.2014.04.020 · Cited by 149 · see citing works
Peng (2018) · doi:10.1145/3214277 · Cited by 134 · see citing works
Mind the Gap: a generative approach to interpretable feature selection and extraction
Kim, B., Shah, J., & Doshi‐Velez, F. (2015) · Neural Information Processing Systems · Cited by 92 · see citing works
Rehabilitation practice: challenges to effective team working
Long, A., Kneafsey, R., & Ryan, J. (2003) · International Journal of Nursing Studies · doi:10.1016/s0020-7489(03)00015-4 · Cited by 85 · see citing works
Yang, Q., Steinfeld, A. M., & Zimmerman, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09612 · Cited by 80 · see citing works
Learning to assess the quality of stroke rehabilitation exercises
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Badia, S. B. I. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302273 · Cited by 71 · see citing works
Self-citationDisagreement in Patient and Carer Assessment of Functional Abilities After Stroke
Knapp, P., & Hewison, J. (1999) · Stroke · doi:10.1161/01.str.30.5.934 · Cited by 59 · see citing works
Movement Quantification in Neurological Diseases: Methods and Applications
Vilas-Boas, M. D. C., & Cunha, J. P. S. (2016) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2016.2543683 · Cited by 41 · see citing works
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A. J. M., & Badia, S. B. I. (2020) · ACM Conference on Health, Inference, and Learning (CHIL) · doi:10.1145/3368555.3384452 · Cited by 37 · see citing works
Self-citationStandards for DSM-5 Reliability
Spitzer, R. L., Williams, J. B. W., & Endicott, J. (2012) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2012.12010083 · Cited by 31 · see citing works
Mispredictions and misrecollections: Challenges for subjective outcome measurement
Smith, D., Brown, S. L., & Ubel, P. A. (2008) · Disability and Rehabilitation · doi:10.1080/09638280701625237 · Cited by 29 · see citing works
An Exploratory Study on Techniques for Quantitative Assessment of Stroke Rehabilitation Exercises
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A. J. M., & Badia, S. B. I. (2020) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3340631.3394872 · Cited by 24 · see citing works
Self-citationLee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A. J. M., & Badia, S. B. I. (2020) · IEEE RO-MAN · doi:10.1109/ro-man47096.2020.9223462 · Cited by 23 · see citing works
Self-citationClassification with Costly Features Using Deep Reinforcement Learning
Janisch, J., Pevný, T., & Lisý, V. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013959 · Cited by 18 · see citing works
An Intelligent Decision Support System for Stroke Rehabilitation Assessment
Lee, M. H. (2019) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3308561.3356106 · Cited by 9 · see citing works
Self-citationAstakhov, V. (2009) · Methods in molecular biology · doi:10.1007/978-1-59745-524-4 · Cited by 7 · see citing works
Intelligent Agent for Assessing and Guiding Rehabilitation Exercises
Lee, M. H. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/911 · Cited by 7 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 6
- [5]5. Mark T Bayley, Amanda Hurdowar, Carol L Richards, Nicol Korner-Bitensky, Sharon Wood-Dauphinee, Janice J Eng, Marilyn McKay-Lyons, Edward Harrison, Robert Teasell, Margaret Harrison, et al. 2012. Barriers to implementation of stroke rehabilitation evidence: findings from a multi-site pilot project. Disability and rehabilitation 34, 19 (2012), 1633-1638.
- [6]6. Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In IJCAI-17 workshop on explainable AI (XAI), Vol. 8. 1.
- [22]22. Henk T Hendricks, Jacques van Limbeek, Alexander C Geurts, and Machiel J Zwarts. 2002. Motor recovery after stroke: a systematic review of the literature. Archives of physical medicine and rehabilitation 83, 11 (2002), 1629-1637.
- [41]41. Rui CV Loureiro, William S Harwin, Kiyoshi Nagai, and Michelle Johnson. 2011. Advances in upper limb stroke rehabilitation: a technology push. Medical & biological engineering & computing 49, 10 (2011), 1103.
- [58]58. Jiliang Tang, Salem Alelyani, and Huan Liu. 2014. Feature selection for classification: A review. Data classification: Algorithms and applications (2014), 37.
- [59]59. Edward Taub, David M Morris, Jean Crago, Danna Kay King, Mary Bowman, Camille Bryson, Staci Bishop, Sonya Pearson, and Sharon E Shaw. 2011. Wolf motor function test (WMFT) manual. Birmingham: University of Alabama, CI Therapy Research Group (2011).
Cited by
Sorted by citations
Chen, J., Yi, C., Okegbile, S. D., Cai, J., & Shen, X. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3308717 · Cited by 268 · see citing works
The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice
Delgado, F., Yang, S., Madaio, M., & Yang, Q. (2023) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3617694.3623261 · Cited by 256 · see citing works
How to Evaluate Trust in AI-Assisted Decision Making? A Survey of Empirical Methodologies
Vereschak, O., Bailly, G., & Caramiaux, B. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476068 · Cited by 229 · see citing works
Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204 · see citing works
A Human-AI Collaborative Approach for Clinical Decision Making on Rehabilitation Assessment
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445472 · Cited by 137 · see citing works
Self-citationHou, Y. T.-Y., & Jung, M. F. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479864 · Cited by 108 · see citing works
AI-Driven Stroke Rehabilitation Systems and Assessment: A Systematic Review
Rahman, S., Sarker, S. C., Haque, A. K. M. N., Uttsha, M. M., Islam, M. F., & Deb, S. (2022) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2022.3219085 · Cited by 89 · see citing works
Lee, M. H., & Chew, C. J. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610218 · Cited by 78 · see citing works
Self-citationVerma, H., Mlynář, J., Schaer, R., Reichenbach, J., Jreige, M., Prior, J. O., Évéquoz, F., & Depeursinge, A. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581506 · Cited by 63 · see citing works
Virtual Reality Solutions Employing Artificial Intelligence Methods: A Systematic Literature Review
de Oliveira, T. R., Rodrigues, B. B., da Silva, M. M., Spinasse, R. A. N., Ludke, G. G., Gaudio, M. R. S., Gomes, G. I. R., Cotini, L. G., Vargens, D. D. S., Schimidt, M. Q., et al. (2022) · ACM Computing Surveys · doi:10.1145/3565020 · Cited by 73 · see citing works
Power and Public Participation in AI
Corbett, E., Denton, E., & Erete, S. L. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623228 · Cited by 57 · see citing works
Ma, S., Wang, X., Lei, Y., Shi, C., Yin, M., & Ma, X. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642671 · Cited by 65 · see citing works
Jesso, S. T., Kelliher, A., Sanghavi, H. K., Martin, T., & Parker, S. H. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.830345 · Cited by 59 · see citing works
Al-Ansari, N., Al‐Thani, D., & Al-Mansoori, R. S. (2024) · Human Behavior and Emerging Technologies · doi:10.1155/2024/4628855 · Cited by 52 · see citing works
A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making
Salimzadeh, S., He, G., & Gadiraju, U. (2023) · Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3565472.3592959 · Cited by 42 · see citing works
Salimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41 · see citing works
“These mechanisms include providing interpretable explanations for AI advice [21, 72, 123], using cognitive forcing functions[17, 47, 99], and incorporating feedback loops to enhance the interaction between humans and AI systems [10, 11, 139].”
Assessing Human Motion During Exercise Using Machine Learning: A Literature Review
Frangoudes, F., Matsangidou, M., Schiza, E. C., Neokleous, K., & Pattichis, C. S. (2022) · IEEE Access · doi:10.1109/access.2022.3198935 · Cited by 41 · see citing works
Ayobi, A., Stawarz, K., Katz, D., Marshall, P., Yamagata, T., Santos‐Rodríguez, R., Flach, P. A., & O’Kane, A. A. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479601 · Cited by 40 · see citing works
Understanding Human-AI Collaboration in Music Therapy Through Co-Design with Therapists
Sun, J., Yang, J., Zhou, G., Jin, Y., & Gong, J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642764 · Cited by 33 · see citing works
Evans, R. P., Bryant, L. D., Russell, G., & Absolom, K. L. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105342 · Cited by 38 · see citing works
Co-design for stroke intervention development: Results of a scoping review
Singh, H., Benn, N., Fung, A., Kokorelias, K. M., Martyniuk, J., Nelson, M. L. A., Colquhoun, H. L., Cameron, J. I., Munce, S., Saragosa, M., et al. (2024) · PLoS ONE · doi:10.1371/journal.pone.0297162 · Cited by 27 · see citing works
Spataru, A., van Dommelen, P., Arnaud, L., Masne, Q. L., Quarteroni, S., & Koledova, E. B. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-01918-2 · Cited by 18 · see citing works
Question-Driven Design Process for Explainable AI User Experiences
Liao, Q. V., Pribić, M., Han, J., Miller, S., & Sow, D. (2021) · arXiv · doi:10.48550/arxiv.2104.03483 · Cited by 18 · see citing works
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511112 · Cited by 19 · see citing works
Self-citationJo, E., Ryu, M., Kenderova, G., So, S., Shapiro, B. B., Papoutsaki, A., & Epstein, D. A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502206 · Cited by 15 · see citing works
Zhang, S., Han, Q., Wang, P., & Li, J. (2025) · Scientific Reports · doi:10.1038/s41598-025-12020-8 · Cited by 16 · see citing works
Jung, H.-T., Kim, Y., Lee, J., Lee, S. I., & Choe, E. K. (2022) · PLoS ONE · doi:10.1371/journal.pone.0274142 · Cited by 14 · see citing works
Stankovics, P., Schillaci, C., Pump, J., Birli, B., Ferraro, G., Munafò, M., Leginio, M. D., Hermann, T., Montanarella, L., & Tóth, G. (2024) · Land Degradation and Development · doi:10.1002/ldr.5030 · Cited by 13 · see citing works
Systematic Review of Virtual Reality Solutions Employing Artificial Intelligence Methods
de Oliveira, T. R., da Silva, M. M., Spinasse, R. A. N., Ludke, G. G., Gaudio, M. R. S., Gomes, G. I. R., Cotini, L. G., Vargens, D. D. S., Schimidt, M. Q., Andreão, R. V., et al. (2021) · Symposium on Virtual and Augmented Reality (SVR) · doi:10.1145/3488162.3488209 · Cited by 15 · see citing works
Using Sensor-Captured Patient-Generated Data to Support Clinical Decision-making in PTSD Therapy
Evans, H. I., Ryu, M., Hsieh, T., Zhou, J., Xu, K., Akers, K. W., Sherrill, A. M., & Arriaga, R. I. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637426 · Cited by 12 · see citing works
Secondary Stakeholders in AI: Fighting for, Brokering, and Navigating Agency
Ajmani, L., Abdelkadir, N. A., & Chancellor, S. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732071 · Cited by 11 · see citing works
Trust in Clinical AI: Expanding the Unit of Analysis
Browne, J., Bakker, S., Yu, B., Lloyd, P. A., & Allouch, S. B. (2022) · Frontiers in artificial intelligence and applications · doi:10.3233/faia220192 · Cited by 10 · see citing works
MR Microsurgical Suture Training System with Level-Appropriate Support
Tashiro, Y., Miyafuji, S., Kojima, Y., Kiyofuji, S., Kin, T., Igarashi, T., & Koike, H. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642324 · Cited by 6 · see citing works
Schäfer, U., Sipos, L., & Müller-Birn, C. (2025) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3757451 · Cited by 11 · see citing works
Kadambi, A., Bandini, A., Ramkalawan, R. D., Hitzig, S. L., & Zariffa, J. (2023) · Topics in Spinal Cord Injury Rehabilitation · doi:10.46292/sci23-00015s · Cited by 6 · see citing works
Raihan, M. J., Ahad, M. A. R., & Nahid, A.-A. (2025) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-025-03372-4 · Cited by 5 · see citing works
Exploring the impact of explainable AI and cognitive capabilities on users’ decisions
Cau, F. M., & Spano, L. D. (2025) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-025-09438-0 · Cited by 9 · see citing works
Towards Uncertainty Aware Task Delegation and Human-AI Collaborative Decision-Making
Lee, M. H., & Tok, M. Z. Y. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732155 · Cited by 5 · see citing works
Self-citationDesign Principles for Human-Centred Explainable AI: A Scoping Review
Hughes, N. G. J., Jia, Y., Sujan, M. A., Lawton, T., Habli, I., & McDermid, J. A. (2025) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3771720 · Cited by 3 · see citing works
Persson, D. R., & Andersen, T. O. (2026) · Digital Health · doi:10.1177/20552076261425134 · Cited by 5 · see citing works
Design And Analysis of Human Computer Interaction Using AI Intelligence
Sivakumar, N., Kiran Kumar, C., Easwaran, B., & Tabassum, H. (2023) · International Conference on Disruptive Technologies (ICDT) · doi:10.1109/icdt57929.2023.10150705 · Cited by 3 · see citing works
Human-AI Teaming: Following the IMOI Framework
Kleanthous, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60611-3_27 · Cited by 3 · see citing works
Context-AI Tunes: Context-Aware AI-Generated Music for Stress Reduction
Wei, X., Zhang, Z., Yue, Z., & Chen, H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93418-6_22 · Cited by 2 · see citing works
Filippetti, M., Censo, R. D., Arcari, L., Schiavariello, M. C., Battaglia, M., Facciorusso, S., Spina, S., Antonucci, L., Santamato, A., Baricich, A., et al. (2025) · Toxins · doi:10.3390/toxins17120573 · Cited by 2 · see citing works
Classification with costly features in hierarchical deep sets
Janisch, J., Pevný, T., & Lisý, V. (2024) · Machine Learning · doi:10.1007/s10994-024-06565-4 · Cited by 2 · see citing works
The role of service design in designing and developing AI applications: Scoping review
Tan, Y. E. I., & Lee, J.-J. (2023) · International Association of Societies of Design Research Congress (IASDR) · doi:10.21606/iasdr.2023.758 · Cited by 1 · see citing works
Li, L., Wang, J., Chen, J., Sun, S., & Peng, W. (2025) · Frontiers in Neuroscience · doi:10.3389/fnins.2025.1737407 · Cited by 1 · see citing works
A Systematic Mapping Study on the Joint Use of AI and VR in Stroke Care
Ferrufino, D., Viado, L., Fronchetti, F., Falcão, D., & Spinola, R. O. (2025) · ACM Symposium on Virtual Reality Software and Technology (VRST) · doi:10.1145/3756884.3766058 · Cited by 1 · see citing works
Perceive, Prioritize, Act: Eliciting Alert Design Requirements for Fast-Paced Medical Work
Zhang, Z., Agbugba, V. F., Billings, S., Bai, E., Luo, X., & Adelgais, K. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2682917 · Cited by 1 · see citing works
SUV: Students’ Understanding Visualizer to Support Instructors in Synchronous Online Lectures
Na, S., Park, S., Jun, J., Jeong, H., Yoo, T., Chun, M., Lee, H., Lee, H. G., & Jung, H. (2023) · IEEE Access · doi:10.1109/access.2023.3303030
Rambach, T., Gleim, P., Heizmann, C., Kellmeyer, P., & Kunze, C. (2025) · doi:10.2196/preprints.88368
Participatory Design of AI-Based Clinical Decision Support Systems: Scoping Review
Rambach, T., Gleim, P., Heizmann, C., Kellmeyer, P., & Kunze, C. (2026) · JMIR Medical Informatics · doi:10.2196/88368
Cabitza, F., Gallazzi, E., & Papale, A. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816965
Rethinking Objectivity in Clinical AI: A Qualitative Study of Concussion Evaluators
Zaidi, A. M., Saw, J. J.-W., Fu, L., Arneson, K., Kim, I., Cross, A., & Karahalios, K. G. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816992
Brown, E. V. D., Lee, M. H., Stepansky, K. E., Fairchild, H., MIKULA, K., Vecchio, M. D., & Halyama, R. (2026) · Work · doi:10.1177/10519815261492235
Self-citationGRAFF-MM : Graph-Frequency Filtering for Multi-Modal Memorability PredictionPreprint
Banna, T. T., Deb, S., Rahman, S., & Rahman, S. (2026) · Research Square · doi:10.21203/rs.3.rs-10283943/v1
- Authors
- Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S.
- Affiliations
- Carnegie Mellon University (US)
- Instituto Superior Técnico (PT)
- University of Madeira, Madeira-ITI, NOVA-LINCS, Funchal, Portugal
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2020-10-14
- Volume
- 4
- Issue
- CSCW2
- Pages
- 1-27
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3094262373
- Semantic Scholar
- 5dce75da3654b5e12ba678fb99f253ab771daae2
- Cited by
- 57
- References
- 61
- Citations by year
- 2026: 7 · 2025: 12 · 2024: 11 · 2023: 11 · 2022: 10 · 2021: 6
- FWCI
- 3.75
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, other-oa)
- Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) (submittedVersion, cc-by-nc-nd)
- Funders
- FCT (SFRH/BD/113694/2015, UIDB/50009/2020)
- ERDF through LISBOA 2020 and FCT (LISBOA-01-0247-FEDER-045948)
- National Science Foundation (CNS-1518865)
- Topics
- Health Sciences › Medicine › Rehabilitation › Stroke Rehabilitation and Recovery
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- clinical decision support systems, stroke rehabilitation, rehabilitation assessment, therapist workload, patient-specific analysis, reinforcement learning, human-centered design, system adoption
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
