Expert Discussions Improve Comprehension of Difficult Cases in Medical Image Assessment
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Medical data labeling workflows critically depend on accurate assessments from human experts. Yet human assessments can vary markedly, even among medical experts. Prior research has demonstrated benefits of labeler training on performance. Here we utilized two types of labeler training feedback: highlighting incorrect labels for difficult cases ("individual performance" feedback), and expert discussions from adjudication of these cases. We presented ten generalist eye care professionals with either individual performance alone, or individual performance and expert discussions from specialists. Compared to performance feedback alone, seeing expert discussions significantly improved generalists' understanding of the rationale behind the correct diagnosis while motivating changes in their own labeling approach; and also significantly improved average accuracy on one of four pathologies in a held-out test set. This work suggests that image adjudication may provide benefits beyond developing trusted consensus labels, and that exposure to specialist discussions can be an effective training intervention for medical diagnosis.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures







Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Individual-level bias can be addressed via expert discussion 145 and labeling adjudication 146 .”
From Deep learning-enabled medical computer vision (2021) · cites this paper as [145]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
Ting, D. S. W., Cheung, C. Y.-L., Lim, G., Tan, G. S. W., Quang, N. D., Gan, A., Hamzah, H., Garcia-Franco, R., San Yeo, I. Y., Lee, S. Y., et al. (2017) · JAMA · doi:10.1001/jama.2017.18152 · Cited by 2,402
Prevalence of Diabetic Retinopathy in the United States, 2005-2008
Zhang, X., Saaddine, J. B., Chou, C.-F., Cotch, M. F., Cheng, Y. J., Geiss, L. S., Gregg, E. W., Albright, A. L., Klein, B. E. K., & Klein, R. (2010) · JAMA · doi:10.1001/jama.2010.1111 · Cited by 1,196
Ting, D. S. W., Cheung, G. C. M., & Wong, T. Y. (2016) · Clinical & Experimental Ophthalmology · doi:10.1111/ceo.12696 · Cited by 1,066
Dunning, D. (2011) · Advances in Experimental Social Psychology · doi:10.1016/b978-0-12-385522-0.00005-6 · Cited by 806
Cardiologist-Level Arrhythmia Detection with Convolutional Neural Networks
Rajpurkar, P., Hannun, A., Haghpanahi, M., Bourn, C., & Ng, A. Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.01836 · Cited by 595
Srinivasan, M., Wilkes, M., Stevenson, F., Nguyen, T., & Slavin, S. (2007) · Academic Medicine · doi:10.1097/01.acm.0000249963.93776.aa · Cited by 569
Krause, J., Gulshan, V., Rahimy, E., Karth, P., Widner, K., Corrado, G. S., Peng, L., & Webster, D. R. (2018) · Ophthalmology · doi:10.1016/j.ophtha.2018.01.034 · Cited by 564
The American Academy of Sleep Medicine Inter-scorer Reliability Program: Sleep Stage Scoring
Rosenberg, R. S., & Van Hout, S. (2013) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.2350 · Cited by 456
Sayres, R., Taly, A., Rahimy, E., Blumer, K., Coz, D., Hammel, N., Krause, J., Narayanaswamy, A., Rastegar, Z., Wu, D., et al. (2019) · Ophthalmology · doi:10.1016/j.ophtha.2018.11.016 · Cited by 418
Self-citation
Research works printed but not linked: 12
- 10.1136/bjophthalmol-2012--302088 · doi:10.1136/bjophthalmol-2012--302088
- 10.5555/3306127.3331890 · doi:10.5555/3306127.3331890
- 10.1111/j.1365--2929.2004.01914.x · doi:10.1111/j.1365--2929.2004.01914.x
- Ryan Drapeau, Lydia B. Chilton, Jonathan Bragg, and Daniel S. Weld. 2016. MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy. In Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP).
- 10.1002/1097-0142(20001015)89: · doi:10.1002/1097-0142(20001015)89
- 10.1016/s0009--9260(03)00063--1 · doi:10.1016/s0009--9260(03)00063--1
- 10.1038/s41562-017-0273--4 · doi:10.1038/s41562-017-0273--4
- 10.1038/s41746-019-0099--8 · doi:10.1038/s41746-019-0099--8
- Mike Schaekermann. 2016. Resolvable vs . Irresolvable Ambiguity : A New Hybrid Framework for Dealing with Uncertain Ground Truth. In ACM SIGCHI Workshop on Human-Centered Machine Learning, Marco Gillies and Rebecca Fiebrink (Eds.). ACM, San Jose. http://hcml2016.goldsmithsdigital.com/program/
- Mike Schaekermann, Edith Law, Kate Larson, and Andrew Lim. 2018. Expert Disagreement in Sequential Labeling: A Case Study on Adjudication in Medical Time Series Analysis. In 1st Workshop on Subjectivity, Ambiguity and Disagreement in Crowdsourcing at HCOMP 2018. Zurich, Switzerland.
- 10.1136/bjophthalmol-2014--305631 · doi:10.1136/bjophthalmol-2014--305631
- 10.1038/s41467-018-07229--3 · doi:10.1038/s41467-018-07229--3
Cited by
Deep learning-enabled medical computer vision
Esteva, A., Chou, K., Yeung, S., Naik, N., Madani, A., Mottaghi, A., Liu, Y., Topol, E., Dean, J., & Socher, R. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00376-2 · Cited by 1,406
BreastScreening-AI: Evaluating medical intelligent agents for human-AI interactions
Calisto, F. M., Santiago, C., Nunes, N., & Nascimento, J. C. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102285 · Cited by 125
Calisto, F. M., Santiago, C., Nunes, N., & Nascimento, J. C. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2021.102607 · Cited by 117
González-Gonzalo, C., Thee, E. F., Klaver, C. C. W., Lee, A. Y., Schlingemann, R. O., Tufail, A., Verbraak, F., & Sánchez, C. I. (2022) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2021.101034 · Cited by 103
“Through surveys and qualitative feedback from graders, Schaekermann et al. (2020) showed that adju-dication provided an effective training intervention, as it significantly C. Gonz ´ alez-Gonzalo et al. improved the graders ’ accuracy and understanding of the rationale behind the correct diagnosis.”
Harnessing Biomedical Literature to Calibrate Clinicians’ Trust in AI Decision Support Systems
Yang, Q., Hao, Y., Quan, K., Yang, S., Zhao, Y., Kuleshov, V., & Wang, F. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581393 · Cited by 72
Metadata
- Authors
- Schaekermann, M., Cai, C. J., Huang, A. E., & Sayres, R.
- Affiliations
- Google (United States) (US)
- Venue
- Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2020-04-21
- Pages
- 1-13
- OpenAlex
- W3009312460
- Cited by
- 28
- References
- 37
- Citations by year
- 2026: 3 · 2025: 5 · 2023: 4 · 2022: 5 · 2021: 10 · 2020: 1
- FWCI
- 5.47
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- Funders
- Google LLC
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- expert discussion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Expert Discussions Improve Comprehension of Difficult Cases in Medical Image Assessment. https://app.cochat.ai/papers/10.1145/3313831.3376290
- The paper
- Schaekermann, M., Cai, C. J., Huang, A. E., & Sayres, R. (2020). Expert Discussions Improve Comprehension of Difficult Cases in Medical Image Assessment. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3313831.3376290
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.