The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy
Open accessAbstract
Deep learning algorithms promise to improve clinician workflows and patient outcomes. However, these gains have yet to be fully demonstrated in real world clinical settings. In this paper, we describe a human-centered study of a deep learning system used in clinics for the detection of diabetic eye disease. From interviews and observation across eleven clinics in Thailand, we characterize current eye-screening workflows, user expectations for an AI-assisted screening process, and post-deployment experiences. Our findings indicate that several socio-environmental factors impact model performance, nursing workflows, and the patient experience. We draw on these findings to reflect on the value of conducting human-centered evaluative research alongside prospective evaluations of model accuracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, many empirical studies on human-AI decision-making did not observe complementary performance [6,16,37,52,58,61,68,82,84,87].”
From Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations (2023) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Clinical usefulness depends on calibration, interpretability, and integration into existing workflows [23,24].”
From Artificial Intelligence in Clinical Decision-Making: A Comprehensive Review of Diagnostic, Prognostic, and Therapeutic Applications, Validation Gaps, Bias, and Deployment Challenges (2026) · cites this paper as [23]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Such distrust has been shown to challenge the adoption of machine learning–based CDSSs [7-10]”
From Factors influencing clinician trust in predictive clinical decision support systems for in-hospital deterioration: qualitative descriptive study (2022) · cites this paper as [8]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“there has been increasing recent research efforts [8, 15, 22, 41, 58, 65] that highlight the importance of involving stakeholders to understand their practices and needs [15, 41, 65] and socio-environmental factors [8] for the design and evaluation of a system.”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [8]
The cited source says what the claim attributes to it.
In the citing paper
“Adoption in clinical practice is influenced by clinician interaction with AI systems, including trust and workflow compatibility [23,27].”
In the citing paper
“For instance, regions of high-throughput healthcare, such as India and Thailand, have welcomed the deployment of technologies such as diabetic retinopathy screening systems 38 .”
From Deep learning-enabled medical computer vision (2021) · cites this paper as [38]
In the citing paper
“For example, when launching ML-powered diabetic retinopathy models in Thailand and India, researchers noticed that model performance was impacted by socioeconomic factors 38 , and determined that where a model is most useful may not be where the model was generated.”
From Deep learning-enabled medical computer vision (2021) · cites this paper as [38]
In the citing paper
“Many studies have investigated the role of AI in assisting humans in making individual decisions [9,16,36, 37, 50, 51, 55, 58, 69, 83, 84, 89], e.g., predicting whether a person will recidivate in the near future.”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [9]
In the citing paper
“AI performance can signifcantly drop for out-of-distribution examples that are diferent from the training data (in-distribution examples) [9,24,46,61].”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
Contextual Design: Defining Customer-Centered Systems
Beyer, H. R., & Holtzblatt, K. (1997) · Cited by 2,748
Abràmoff, M. D., Lavin, P. T., Birch, M., Shah, N., & Folk, J. C. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0040-6 · Cited by 1,638
Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594
Self-citationElwyn, G. J., Scholl, I., Tietbohl, C. K., Mann, M. K., Edwards, A., Clay, C. F., Légaré, F., van der Weijden, T., Lewis, C. L., Wexler, R., et al. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-s2-s14 · Cited by 515
Diabetes patient education: a meta-analysis and meta-regression
Ellis, S. E., Speroff, T., Dittus, R. S., Brown, A. W., Pichert, J. W., & Elasy, T. A. (2003) · Patient Education and Counseling · doi:10.1016/s0738-3991(03)00016-8 · Cited by 492
A Review of 25 Years of CSCW Research in Healthcare: Contributions, Challenges and Future Agendas
Fitzpatrick, G., & Ellingsen, G. (2012) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-012-9168-0 · Cited by 439
Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405
Participatory IT Design: Designing for Business and Workplace Realities
Bødker, K., Kensing, F., & Simonsen, J. (2004) · Cited by 346
Giger, M. L., Chan, H., & Boone, J. (2008) · Medical Physics · doi:10.1118/1.3013555 · Cited by 311
Flagged first, then by citations
Research works printed but not linked: 3
- Kathleen Musante DeWalt and Billie R DeWalt. 2002. Participant Observation: A Guide for Fieldworkers. Rowman Altamira.
- Jenchitr W, Hanutsaha P, Iamsirithaworn S, Parnrat U, Choosri P. 2007. The national survey of blindness low vision and visual impairment in thailand 2006--2007. Thai J Pub Hlth Ophthalmol 21, 1 (2007), 10.
- Paisan Raumviboonsuk, Jonathan Krause, Peranut Chotcomwongse, Rory Sayres, Rajiv Raman, Kasumi Widner, Bilson JL Campana, Sonia Phene, Kornwipa Hemarat, Mongkol Tadarati, and others. 2019. Deep learning versus human graders for classifying diabetic retinopathy severity in a nationwide screening program. npj Digital Medicine 2, 1 (2019), 25.
Cited by
Sorted by citations
Items 601–639 of 639
Barrientos, K. M., Villa, M. C., Castano-Villegas, N., Zea, J., & Velásquez, L. (2026) · medRxiv · doi:10.64898/2026.06.26.26356340
Sarkar, J., Drovandi, C., & Reddy, S. (2026) · JMIR Medical Informatics · doi:10.2196/89113
Human–GenAI collaboration: a framework for engagement, self-regulation and ethical learning outcomes
Jianhong, L., Agarwal, V., Alzeiby, E. A., Dernóczi-Polyák, A., & Tuân, N. P. (2026) · Journal of Enterprise Information Management · doi:10.1108/jeim-11-2025-1148
Kumar, A., Mala, D. J., Sai, A., & Guleria, A. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3911
Li, D., Heman, Z., Lu, C., & Lu, C. (2026) · Digital Health · doi:10.1177/20552076261477199
Exploratory Study of the Determinants of Medical AI Adoption
Chae, J.-B., & Kim, K.-J. (2026) · The Korean Journal of Health Service Management · doi:10.12811/kshsm.2026.20.2.031
Quality Assessment, Enhancement and Active Retraining for Reliable Fundus Image Analysis
Babu, K. D., Yohannan, R. P., & Sumam David, S. (2026) · International Conference on Electronics, Computing, Communication and Control Technology (ICECCC) · doi:10.1109/iceccc70334.2026.11633224
Belghachi, D., & Soussi, L. H. L. (2026) · International Conference on Circuit, Systems and Communication (ICCSC) · doi:10.1109/iccsc71566.2026.11650469
Causal attributions for ChatGPT errors: what is the role of AI literacy?
Norcia, M., & Colí, E. (2026) · Journal of Psychology and AI · doi:10.1080/29974100.2026.2712302
Integration of artificial intelligence and multi-omics for precision medicine
Marei, H. E. S., Cenciarelli, C., & Vagni, D. (2026) · Functional & Integrative Genomics · doi:10.1007/s10142-026-02024-6
From Labels to Language: Zero-Shot Radiology Report Generation with Hybrid Retrieval Augmentation
Nishizawa, T., Issah, A., Fuertes-Brito, A., Manzoor, K., Jiao, Z., & Qamar, N. (2026) · IEEE/ACM Conference on Connected Health: Applications, Systems and Engineering Technologies · doi:10.1109/chase69719.2026.00062
Decision substitution and the reorganisation of judgement in AI-mediated society
Ibitoye, A., Ajenifuja, O. F., Kareem, R. S. A., Adeyemo, D. D., & Famutimi, R. F. (2026) · AI & Society · doi:10.1007/s00146-026-03349-3
A Reinforcement-Governed Treatment Policy Architecture for Clinical Workflow Integration
Martins, B., Pereira, L., Azevedo, R., & Costa, P. (2023) · Journal of Artificial Intelligence for Healthcare Systems · doi:10.68159/g774254179
A Large Language Model Integration Architecture for Clinical Decision Infrastructure
Turner, M., Nguyen, S., Clark, D., & Wilson, E. (2025) · Journal of Artificial Intelligence for Healthcare Systems · doi:10.68159/c376023636
A Transformer-Embedded Clinical Phenotyping Infrastructure Model
Andrade, L., Lopes, M., & Brooks, D. (2024) · Journal of Artificial Intelligence for Healthcare Systems · doi:10.68159/i117109956
Sebin, B., & Sebin, I. K. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1956533
How to apply artificial intelligence (AI) to facilitate and enhance MASH trials
Lim, D. Y. Z., Leow, W. Q., Allende, D. S., Seto, W., Yip, T. C., & Goh, G. B. (2026) · Hepatology International · doi:10.1007/s12072-026-11150-z
xAARA: Explainable and Augmented Automated Rehabilitation Assessment
Ahmed, T., Rikakis, T., Kelliher, A., Khan, M. S. S., & Guo, Z. (2026) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3848511
Yang, M., Park, C., Won, H., Kim, T., Song, H., & Han, J. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816903
Brown, N., Taylor, E., Wilson, D. J., & Scott, O. (2026) · Journal of Health Informatics and Digital Systems · doi:10.68159/q639063208
Ontika, N. N., Saßmannshausen, S. M., Rohde, M., & de Carvalho, A. F. P. (2026) · Nordic Conference on Human-Computer Interaction · doi:10.1145/3829807.3829885
Jin, K., Zhao, K., Agrawal, R. V., Ying, G., & Grzybowski, A. E. (2026) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2026.101533
AI Prognosis in Healthcare, Bias, Misprediction, and Governance Gaps
Düffelmeyer, M. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-0403-6.ch008
Castano-Villegas, N., Barrientos, K. M., Zea, J., & Velásquez, L. (2026) · doi:10.2196/preprints.113693
Lesion-Aware Multi-Task Learning with Explainable AI for Retinal Disease Classification
Aryan, H., -, K. E. R., Raj, S. R., & Jabbar, M. A. (2026) · International Conference on Natural Language Processing and Computer Vision (ICNPCV) · doi:10.1109/icnpcv69434.2026.11708994
RELY-SE: A Dashboard for Measuring Operational Reliability of AI Coding Agents
Akilesh, P., Shivadharshan, S., Chimalakonda, S., Sharma, V. S., & Kaulgud, V. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834654
Alzubaidi, M. S., Muaz, R., Abbes, I., Alyafei, K. A., Househ, M., & Agus, M. (2026) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms69103.2026.00116
Adapa, K., Garg, M., Sethi, R., Venketa, A. Y., Chauhan, T., Regi, M., & Mazur, Ł. (2026) · Asia-Pacific Journal of Oncology Nursing · doi:10.1016/j.apjon.2026.101044
An all-in-one foundation model for multimodal medical image restoration and enhancement
Cao, L., Wu, Z., Zhang, J., Qi, L., Gao, Y., Zheng, H., Zheng, Y., & Shi, Y. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-026-01310-z
Karpov, O. E., Осипов, А. В., Пензин, О. В., & Soultati, A. (2026) · Vrach i informacionnye tehnologii · doi:10.25881/18110193_2026_3_6
Li, R., Ma, T., Tang, L., Li, S., Wu, Y., & Xia, H. (2026) · medRxiv · doi:10.64898/2026.10.23.26366103
Explainable Multi-Agent Clinical Intelligence Framework for Advanced Healthcare Decision Support
Valiki, D. (2026) · International Journal of Modern Scientific Discoveries · doi:10.64137/31079377/ijmsd-v2i3p103
Oladele, M. O., Alimi, O. A., & Olugbara, O. O. (2026) · Diagnostics · doi:10.3390/diagnostics16213546
Menna, F., Meduri, A., Lupo, S., Luca, L. D., Baldascino, A., & Vingolo, E. M. (2026) · Medicina · doi:10.3390/medicina62112099
Deobhakta, A. A., & Tsai, J. C. (2026) · BMJ Connections Digital Health & AI · doi:10.1136/bmjcdh-2026-000031
Rathore, H., & Rodriguez, R. (2026) · doi:10.2196/preprints.115523
Hemşirelikte Yapay Zekâ Uygulamaları: Klinik Kullanım Alanları ve Bakım Süreçleri
Dağlı, İ., & Dağlı, S. (2026) · Özgür Yayınları eBooks · doi:10.58830/ozgur.pub1577.c6483
Metadata
- Authors
- Beede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., & Vardoulakis, L. M.
- Affiliations
- Google (United States) (US)
- Google (Singapore) (SG)
- Rajavithi Hospital (TH)
- Venue
- Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2020-04-21
- Pages
- 1-12
- OpenAlex
- W3004483087
- Cited by
- 614
- References
- 34
- Citations by year
- 2026: 85 · 2025: 115 · 2024: 104 · 2023: 89 · 2022: 97 · 2021: 94 · 2020: 28
- FWCI
- 39.56
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- diabetic retinopathy detection, human-centered evaluation, AI-assisted screening, nursing workflow, patient experience, socio-environmental factors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
