CoChat

Expert Discussions Improve Comprehension of Difficult Cases in Medical Image Assessment

Open access
Schaekermann, Mike, Cai, Carrie J., Huang, Abigail E., Sayres, Rory · 2020 · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.

Abstract

Medical data labeling workflows critically depend on accurate assessments from human experts. Yet human assessments can vary markedly, even among medical experts. Prior research has demonstrated benefits of labeler training on performance. Here we utilized two types of labeler training feedback: highlighting incorrect labels for difficult cases ("individual performance" feedback), and expert discussions from adjudication of these cases. We presented ten generalist eye care professionals with either individual performance alone, or individual performance and expert discussions from specialists. Compared to performance feedback alone, seeing expert discussions significantly improved generalists' understanding of the rationale behind the correct diagnosis while motivating changes in their own labeling approach; and also significantly improved average accuracy on one of four pathologies in a held-out test set. This work suggests that image adjudication may provide benefits beyond developing trusted consensus labels, and that exposure to specialist discussions can be an effective training intervention for medical diagnosis.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3313831.3376290

Cites

42 of 42 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Task interface for medical image assessment. The medical image shown is an illustrative example rather than from the real dataset.
Figure 1 Task interface for medical image assessment. The medical image shown is an illustrative example rather than from the real dataset.CC-BY · From the PDF
Figure 2 Training feedback interface for medical generalists. The medical image shown is an illustrative example rather than from the real dataset.
Figure 2 Training feedback interface for medical generalists. The medical image shown is an illustrative example rather than from the real dataset.CC-BY · From the PDF
Figure 3 Generalists' perception of training feedback.
Figure 3 Generalists' perception of training feedback.CC-BY · From the PDF
Figure 4 Average change in generalists' diagnostic accuracy per casepair in train set and held-out test set. Error bars indicate 95% confidence intervals.
Figure 4 Average change in generalists' diagnostic accuracy per casepair in train set and held-out test set. Error bars indicate 95% confidence intervals.CC-BY · From the PDF
Figure 5 Improvement in generalists' self-efficacy score for diagnosis of retinal artery occlusion (RAO) after training feedback.
Figure 5 Improvement in generalists' self-efficacy score for diagnosis of retinal artery occlusion (RAO) after training feedback.CC-BY · From the PDF
Figure 6 Change in generalists' diagnostic confidence and perceived case difficulty per case-pair in train set and held-out test set.
Figure 6 Change in generalists' diagnostic confidence and perceived case difficulty per case-pair in train set and held-out test set.CC-BY · From the PDF
Figure 7 Example adjudication discussions with mixed ratings (cases A and B) and high ratings (cases C and D) for answer key comprehension.
Figure 7 Example adjudication discussions with mixed ratings (cases A and B) and high ratings (cases C and D) for answer key comprehension.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 42 references

Research works printed but not linked: 12

  1. 10.1136/bjophthalmol-2012--302088 · doi:10.1136/bjophthalmol-2012--302088
  2. 10.5555/3306127.3331890 · doi:10.5555/3306127.3331890
  3. 10.1111/j.1365--2929.2004.01914.x · doi:10.1111/j.1365--2929.2004.01914.x
  4. Ryan Drapeau, Lydia B. Chilton, Jonathan Bragg, and Daniel S. Weld. 2016. MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy. In Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP).
  5. 10.1002/1097-0142(20001015)89: · doi:10.1002/1097-0142(20001015)89
  6. 10.1016/s0009--9260(03)00063--1 · doi:10.1016/s0009--9260(03)00063--1
  7. 10.1038/s41562-017-0273--4 · doi:10.1038/s41562-017-0273--4
  8. 10.1038/s41746-019-0099--8 · doi:10.1038/s41746-019-0099--8
  9. Mike Schaekermann. 2016. Resolvable vs . Irresolvable Ambiguity : A New Hybrid Framework for Dealing with Uncertain Ground Truth. In ACM SIGCHI Workshop on Human-Centered Machine Learning, Marco Gillies and Rebecca Fiebrink (Eds.). ACM, San Jose. http://hcml2016.goldsmithsdigital.com/program/
  10. Mike Schaekermann, Edith Law, Kate Larson, and Andrew Lim. 2018. Expert Disagreement in Sequential Labeling: A Case Study on Adjudication in Medical Time Series Analysis. In 1st Workshop on Subjectivity, Ambiguity and Disagreement in Crowdsourcing at HCOMP 2018. Zurich, Switzerland.
  11. 10.1136/bjophthalmol-2014--305631 · doi:10.1136/bjophthalmol-2014--305631
  12. 10.1038/s41467-018-07229--3 · doi:10.1038/s41467-018-07229--3

Cited by

See all 28

Metadata

Authors
Schaekermann, M., Cai, C. J., Huang, A. E., & Sayres, R.
Affiliations
Google (United States) (US)
Venue
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2020-04-21
Pages
1-13
OpenAlex
W3009312460
Cited by
28
References
37
Citations by year
2026: 3 · 2025: 5 · 2023: 4 · 2022: 5 · 2021: 10 · 2020: 1
FWCI
5.47
Type
conference paper
Language
en
Open access
Locations
CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
Funders
Google LLC
Topics
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
expert discussion

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Expert Discussions Improve Comprehension of Difficult Cases in Medical Image Assessment. https://app.cochat.ai/papers/10.1145/3313831.3376290
The paper
Schaekermann, M., Cai, C. J., Huang, A. E., & Sayres, R. (2020). Expert Discussions Improve Comprehension of Difficult Cases in Medical Image Assessment. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3313831.3376290

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.