Expert Discussions Improve Comprehension of Difficult Cases in Medical Image Assessment
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Medical data labeling workflows critically depend on accurate assessments from human experts. Yet human assessments can vary markedly, even among medical experts. Prior research has demonstrated benefits of labeler training on performance. Here we utilized two types of labeler training feedback: highlighting incorrect labels for difficult cases ("individual performance" feedback), and expert discussions from adjudication of these cases. We presented ten generalist eye care professionals with either individual performance alone, or individual performance and expert discussions from specialists. Compared to performance feedback alone, seeing expert discussions significantly improved generalists' understanding of the rationale behind the correct diagnosis while motivating changes in their own labeling approach; and also significantly improved average accuracy on one of four pathologies in a held-out test set. This work suggests that image adjudication may provide benefits beyond developing trusted consensus labels, and that exposure to specialist discussions can be an effective training intervention for medical diagnosis.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures







Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Individual-level bias can be addressed via expert discussion 145 and labeling adjudication 146 .”
From Deep learning-enabled medical computer vision (2021) · cites this paper as [145]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
Ting, D. S. W., Cheung, C. Y.-L., Lim, G., Tan, G. S. W., Quang, N. D., Gan, A., Hamzah, H., Garcia-Franco, R., San Yeo, I. Y., Lee, S. Y., et al. (2017) · JAMA · doi:10.1001/jama.2017.18152 · Cited by 2,402
Prevalence of Diabetic Retinopathy in the United States, 2005-2008
Zhang, X., Saaddine, J. B., Chou, C.-F., Cotch, M. F., Cheng, Y. J., Geiss, L. S., Gregg, E. W., Albright, A. L., Klein, B. E. K., & Klein, R. (2010) · JAMA · doi:10.1001/jama.2010.1111 · Cited by 1,196
Ting, D. S. W., Cheung, G. C. M., & Wong, T. Y. (2016) · Clinical & Experimental Ophthalmology · doi:10.1111/ceo.12696 · Cited by 1,066
Dunning, D. (2011) · Advances in Experimental Social Psychology · doi:10.1016/b978-0-12-385522-0.00005-6 · Cited by 806
Cardiologist-Level Arrhythmia Detection with Convolutional Neural Networks
Rajpurkar, P., Hannun, A., Haghpanahi, M., Bourn, C., & Ng, A. Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.01836 · Cited by 595
Srinivasan, M., Wilkes, M., Stevenson, F., Nguyen, T., & Slavin, S. (2007) · Academic Medicine · doi:10.1097/01.acm.0000249963.93776.aa · Cited by 569
Krause, J., Gulshan, V., Rahimy, E., Karth, P., Widner, K., Corrado, G. S., Peng, L., & Webster, D. R. (2018) · Ophthalmology · doi:10.1016/j.ophtha.2018.01.034 · Cited by 564
The American Academy of Sleep Medicine Inter-scorer Reliability Program: Sleep Stage Scoring
Rosenberg, R. S., & Van Hout, S. (2013) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.2350 · Cited by 456
Sayres, R., Taly, A., Rahimy, E., Blumer, K., Coz, D., Hammel, N., Krause, J., Narayanaswamy, A., Rastegar, Z., Wu, D., et al. (2019) · Ophthalmology · doi:10.1016/j.ophtha.2018.11.016 · Cited by 418
Self-citationNeural network analysis of sleep stages enables efficient diagnosis of narcolepsy
Stephansen, J. B., Olesen, A. N., Olsen, M., Ambati, A., Leary, E. B., Moore, H. E., Carrillo, O., Lin, L., Han, F., Yan, H., et al. (2018) · Nature Communications · doi:10.1038/s41467-018-07229-3 · Cited by 413
EyePACS: An Adaptable Telemedicine System for Diabetic Retinopathy Screening
Cuadros, J., & Bresnick, G. (2009) · Journal of Diabetes Science and Technology · doi:10.1177/193229680900300315 · Cited by 390 · cited 1×
“The labeling workflow we use here is similar to that used in telemedicine enterprises, including screening for eye disease [9, 5].”
MethodVariability of expert observers in evaluating the optic disc.
Lichter, P. R. (1976) · Europe PMC (PubMed Central) · Cited by 306
How to develop machine learning models for healthcare
Chen, P.-H. C., Liu, Y., & Peng, L. (2019) · Nature Materials · doi:10.1038/s41563-019-0345-0 · Cited by 274
Ruamviboonsuk, P., Krause, J., Chotcomwongse, P., Sayres, R., Raman, R., Widner, K., Campana, B. J. L., Phene, S., Hemarat, K., Tadarati, M., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0099-8 · Cited by 233
Self-citationWho Said What: Modeling Individual Labelers Improves Classification
Guan, M., Gulshan, V., Dai, A., & Hinton, G. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11756 · Cited by 191
Deep Learning and Glaucoma Specialists
Phene, S., Dunn, R. C., Hammel, N., Liu, Y., Krause, J., Kitade, N., Schaekermann, M., Sayres, R., Wu, D. J., Bora, A., et al. (2019) · Ophthalmology · doi:10.1016/j.ophtha.2019.07.024 · Cited by 186 · cited 1×
“Digital fundus photos are used both in tele-medical screening [40] and for the development of deep learning models for AI-assisted retinal assessment [17, 27].”
MethodSelf-citationAggregated knowledge from a small number of debates outperforms the wisdom of large crowds
Navajas, J., Niella, T., Garbulsky, G., Bahrami, B., & Sigman, M. (2018) · Nature Human Behaviour · doi:10.1038/s41562-017-0273-4 · Cited by 183
Telemedicine for detecting diabetic retinopathy: a systematic review and meta-analysis
Shi, L., Wu, H., Dong, J., Jiang, K., Lu, X., & Shi, J. (2015) · British Journal of Ophthalmology · doi:10.1136/bjophthalmol-2014-305631 · Cited by 174
Barnett, M. L., Boddupalli, D., Nundy, S., & Bates, D. W. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.0096 · Cited by 154 · cited 3×
“Several works have suggested ways to make productive use of disagreement information in medical data labels (e.g., [2, 3, 8, 19, 28, 33, 36]).”
BackgroundDirect Uncertainty Prediction for Medical Second Opinions
Raghu, M., Blumer, K., Sayres, R., Obermeyer, Z., Kleinberg, R., Mullainathan, S., & Kleinberg, J. (2018) · arXiv · doi:10.48550/arxiv.1807.01771 · Cited by 93
Self-citationKronz, J. D., Silberman, M. A., Allsbrook, W. C., & Epstein, J. I. (2000) · PubMed · Cited by 86
Plus Disease in Retinopathy of Prematurity
Kalpathy-Cramer, J., Campbell, J. P., Erdogmus, D., Tian, P., Kedarisetti, D., Moleta, C., Reynolds, J. D., Hutcheson, K., Shapiro, M. J., Repka, M. X., et al. (2016) · Ophthalmology · doi:10.1016/j.ophtha.2016.07.020 · Cited by 86
Crowdsourcing Ground Truth for Medical Relation Extraction
Dumitrache, A., Aroyo, L., & Welty, C. (2018) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3152889 · Cited by 81 · cited 1×
“Prior work in human computation for medical relation extraction [13] views inter-rater disagreement as a function of three phenomena: differences among human raters (e.g., training background, biases), characteristics of the data to be labeled (e.g., noisy or heterogeneous), and the quality of the…”
BackgroundClinical teachers and problem‐based learning: a phenomenological study
Dornan, T., Scherpbier, A., King, N., & Boshuizen, H. (2005) · Medical Education · doi:10.1111/j.1365-2929.2004.01914.x · Cited by 61
Teleretinal Imaging to Screen for Diabetic Retinopathy in the Veterans Health Administration
Cavallerano, A. A., & Conlin, P. R. (2008) · Journal of Diabetes Science and Technology · doi:10.1177/193229680800200106 · Cited by 56
Resolvable vs. Irresolvable Disagreement: A Study on Worker Deliberation in Crowd Work
Schaekermann, M., Goh, J., Larson, K., & Law, E. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274423 · Cited by 54
Self-citationThe global inverse care law: a distorted map of blindness
Bastawrous, A., & Hennig, B. D. (2012) · British Journal of Ophthalmology · doi:10.1136/bjophthalmol-2012-302088 · Cited by 47
Understanding Expert Disagreement in Medical Data Analysis through Structured Adjudication
Schaekermann, M., Beaton, G., Habib, M., Lim, A., Larson, K., & Law, E. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359178 · Cited by 47
Self-citationInterictal epileptiform discharge characteristics underlying expert interrater agreement
Bagheri, E., Dauwels, J., Dean, B. C., Waters, C. G., Westover, M. B., & Halford, J. J. (2017) · Clinical Neurophysiology · doi:10.1016/j.clinph.2017.06.252 · Cited by 46
Ambiguity-aware AI Assistants for Medical Data Analysis
Schaekermann, M., Beaton, G., Sanoubari, E., Lim, A., Larson, K., & Law, E. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376506 · Cited by 44
Self-citationTo Re(label), or Not To Re(label)
Lin, C., Mausam, & Weld, D. (2014) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v2i1.13167 · Cited by 42
Cicero: Multi-Turn, Contextual Argumentation for Accurate Crowdsourcing
Chen, Q., Bragg, J., Chilton, L. B., & Weld, D. S. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300761 · Cited by 41
Validation of Consensus Panel Diagnosis in Dementia
Gabel, M. J., Foster, N. L., Heidebrink, J. L., Higdon, R., Aizenstein, H. J., Arnold, S. E., Barbas, N. R., Boeve, B. F., Burke, J. R., Clark, C. M., et al. (2010) · Archives of Neurology · doi:10.1001/archneurol.2010.301 · Cited by 40
Ali, A. A. H. A. A. E. S. B. A., Hallingham, S., & Buys, Y. M. (2015) · Canadian Journal of Ophthalmology · doi:10.1016/j.jcjo.2015.09.001 · Cited by 38
Remote Tool-Based Adjudication for Grading Diabetic Retinopathy
Schaekermann, M., Hammel, N., Terry, M., Ali, T. K., Liu, Y., Basham, B., Campana, B., Chen, W., Ji, X., Krause, J., et al. (2019) · Translational Vision Science & Technology · doi:10.1167/tvst.8.6.40 · Cited by 19 · cited 4×
“While most tele-medical grading centers do not yet use such kind of procedure, there is a growing body of research in HCI on designing and developing methods for online group discussion among crowd workers, that could be leveraged towards a broader applicability of our approach [38].”
BackgroundMethodInfluentialSelf-citationCan the NHS Breast Screening Programme Afford not to Double Read Screening Mammograms?
Liston, J. C., & Dall, B. J. G. (2003) · Clinical Radiology · doi:10.1016/s0009-9260(03)00063-1 · Cited by 18
Diwan, J., Shah, C., Sanghavi, S., & Shah, A. M. (2017) · National Journal of Physiology Pharmacy and Pharmacology · doi:10.5455/njppp.2017.7.0204220032017 · Cited by 15
Hartswood, M., Procter, R., Taylor, P. M., Blot, L., Anderson, S., Rouncefield, M., & Slack, R. (2012) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208533 · Cited by 14
Prostaglandin agonist effect on matrix metalloproteinase aqueous levels in glaucoma patients
Pradhan, Z. S., Dalvi, R. A., Lai, T. F., Kranemann, C. F., Boyd, S. R., & Birt, C. M. (2015) · Canadian Journal of Ophthalmology · doi:10.1016/j.jcjo.2014.10.006 · Cited by 11
Domain-Independent Quality Measures for Crowd Truth Disagreement.
Inel, O., Aroyo, L. M., Welty, C., & Sips, R.-J. (2013) · Cited by 8
Capturing Expert Arguments from Medical Adjudication Discussions in a Machine-readable Format
Schaekermann, M., Beaton, G., Habib, M., Lim, A. S. P., Larson, K. S., & Law, E. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308560.3317085 · Cited by 7
Self-citation
Research works printed but not linked: 12
- 10.1136/bjophthalmol-2012--302088 · doi:10.1136/bjophthalmol-2012--302088
- Robin Cohen, Mike Schaekermann, Sihao Liu, and Michael Cormier. 2019. Trusted AI and the Contribution of Trust Modeling in Multiagent Systems. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS '19). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 1644-1648. http://dl.acm.org/citation.cfm?id=3306127.3331890 · doi:10.5555/3306127.3331890
- 10.1111/j.1365--2929.2004.01914.x · doi:10.1111/j.1365--2929.2004.01914.x
- Ryan Drapeau, Lydia B. Chilton, Jonathan Bragg, and Daniel S. Weld. 2016. MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy. In Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP).
- Joseph D Kronz, Mark A Silberman, William C Allsbrook, and Jonathan I Epstein. 2000. A web-based tutorial improves practicing pathologists' Gleason Paper 163 grading of images of prostate carcinoma specimens obtained by needle biopsy. Cancer 89, 8 (oct 2000), 1818-1823. DOI: http://dx.doi.org/10.1002/1097-0142(20001015)89: 8<1818::AID-CNCR23>3.0.CO;2-J · doi:10.1002/1097-0142(20001015)89
- 10.1016/s0009--9260(03)00063--1 · doi:10.1016/s0009--9260(03)00063--1
- 10.1038/s41562-017-0273--4 · doi:10.1038/s41562-017-0273--4
- 10.1038/s41746-019-0099--8 · doi:10.1038/s41746-019-0099--8
- Mike Schaekermann. 2016. Resolvable vs . Irresolvable Ambiguity : A New Hybrid Framework for Dealing with Uncertain Ground Truth. In ACM SIGCHI Workshop on Human-Centered Machine Learning, Marco Gillies and Rebecca Fiebrink (Eds.). ACM, San Jose. http://hcml2016.goldsmithsdigital.com/program/
- Mike Schaekermann, Edith Law, Kate Larson, and Andrew Lim. 2018. Expert Disagreement in Sequential Labeling: A Case Study on Adjudication in Medical Time Series Analysis. In 1st Workshop on Subjectivity, Ambiguity and Disagreement in Crowdsourcing at HCOMP 2018. Zurich, Switzerland.
- 10.1136/bjophthalmol-2014--305631 · doi:10.1136/bjophthalmol-2014--305631
- 10.1038/s41467-018-07229--3 · doi:10.1038/s41467-018-07229--3
Cited by
Deep learning-enabled medical computer vision
Esteva, A., Chou, K., Yeung, S., Naik, N., Madani, A., Mottaghi, A., Liu, Y., Topol, E., Dean, J., & Socher, R. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00376-2 · Cited by 1,406
BreastScreening-AI: Evaluating medical intelligent agents for human-AI interactions
Calisto, F. M., Santiago, C., Nunes, N., & Nascimento, J. C. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102285 · Cited by 125
Calisto, F. M., Santiago, C., Nunes, N., & Nascimento, J. C. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2021.102607 · Cited by 117
González-Gonzalo, C., Thee, E. F., Klaver, C. C. W., Lee, A. Y., Schlingemann, R. O., Tufail, A., Verbraak, F., & Sánchez, C. I. (2022) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2021.101034 · Cited by 103
“Through surveys and qualitative feedback from graders, Schaekermann et al. (2020) showed that adju-dication provided an effective training intervention, as it significantly C. Gonz ´ alez-Gonzalo et al. improved the graders ’ accuracy and understanding of the rationale behind the correct diagnosis.”
Harnessing Biomedical Literature to Calibrate Clinicians’ Trust in AI Decision Support Systems
Yang, Q., Hao, Y., Quan, K., Yang, S., Zhao, Y., Kuleshov, V., & Wang, F. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581393 · Cited by 72
Metadata
- Authors
- Schaekermann, M., Cai, C. J., Huang, A. E., & Sayres, R.
- Affiliations
- Google (United States) (US)
- Venue
- Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2020-04-21
- Pages
- 1-13
- OpenAlex
- W3009312460
- Cited by
- 28
- References
- 37
- Citations by year
- 2026: 3 · 2025: 5 · 2023: 4 · 2022: 5 · 2021: 10 · 2020: 1
- FWCI
- 5.47
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- Funders
- Google LLC
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- expert discussion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.