How to Discriminate between Computer-Aided and Computer-Hindered Decisions: A Case Study in Mammography
Open accessAbstract
Background. Computer aids can affect decisions in complex ways, potentially even making them worse; common assessment methods may miss these effects. We developed a method for estimating the quality of decisions, as well as how computer aids affect it, and applied it to computer-aided detection (CAD) of cancer, reanalyzing data from a published study where 50 professionals (“readers”) interpreted 180 mammograms, both with and without computer support. Method. We used stepwise regression to estimate how CAD affected the probability of a reader making a correct screening decision on a patient with cancer (sensitivity), thereby taking into account the effects of the difficulty of the cancer (proportion of readers who missed it) and the reader’s discriminating ability (Youden’s determinant). Using regression estimates, we obtained thresholds for classifying a posteriori the cases (by difficulty) and the readers (by discriminating ability). Results. Use of CAD was associated with a 0.016 increase in sensitivity (95% confidence interval [CI], 0.003–0.028) for the 44 least discriminating radiologists for 45 relatively easy, mostly CAD-detected cancers. However, for the 6 most discriminating radiologists, with CAD, sensitivity decreased by 0.145 (95% CI, 0.034–0.257) for the 15 relatively difficult cancers. Conclusions. Our exploratory analysis method reveals unexpected effects. It indicates that, despite the original study detecting no significant average effect, CAD helped the less discriminating readers but hindered the more discriminating readers. Such differential effects, although subtle, may be clinically significant and important for improving both computer algorithms and protocols for their use. They should be assessed when evaluating CAD and similar warning systems.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Responses can vary depending on the task at hand or the person's task expertise; algorithmic appreciation waned when participants had high domain expertise.
“Responses can vary depending on the task at hand or the person’s task expertise—for instance, one study found that algorithmic appreciation waned when the participants had high domain expertise15,17.”
From Do as AI say: susceptibility in deployment of clinical decision-aids (2021) · cites this paper as [17]
The Discussion and Conclusions (page unavailable) says CAD affected readers differently according to their skill and the difficulty of the mammogram; highly discriminating readers had degraded sensitivity on difficult cases. The paper measures decision performance, not participants’ appreciation of algorithms, so it does not establish that appreciation waned among people with high domain expertise. (LLM, conf=0.86)
Experimental — accuracy not yet measuredIn a study of radiologists, decision aid helped lower-performing radiologists but hindered the performance of the best ones.
“Similarly, a study of radiologists showed that while decision aid helped lower-performing radiologists, it actually hindered the performance of the best ones [39].”
From Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiative (2021) · cites this paper as [39]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305 · see citing works
Hosmer, D. W., & Lemeshow, S. (2000) · doi:10.1002/0471722146 · Cited by 20,302 · see citing works
McCullagh, P., & Nelder, J. A. (1989) · doi:10.1007/978-1-4899-3242-6 · Cited by 16,836 · see citing works
Humans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · see citing works
An Introduction to Generalized Linear Models.
Chang, P. C., & Dobson, A. J. (1991) · Journal of the American Statistical Association · doi:10.2307/2290548 · Cited by 2,543 · see citing works
Akaike Information Criterion Statistics.
Grove, D., Sakamoto, Y., Ishiguro, M., & Kitagawa, G. (1988) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2348776 · Cited by 1,184 · see citing works
Automation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · see citing works
Potential Contribution of Computer-aided Detection to the Sensitivity of Screening Mammography
Burhenne, L. J. W., Wood, S. A., D’Orsi, C. J., Feig, S. A., Kopans, D. B., O′Shaughnessy, K. F., Sickles, E. A., Tabár, L. K., Vyborny, C. J., & Castellino, R. A. (2000) · Radiology · doi:10.1148/radiology.215.2.r00ma15554 · Cited by 552 · see citing works
An Introduction to Generalized Linear Models
Dobson, A. J. (1990) · Springer eBooks · doi:10.1007/978-1-4899-7252-1 · Cited by 543 · see citing works
Accountability and automation bias
SKITKA, L. J., MOSIER, K., & BURDICK, M. D. (2000) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0349 · Cited by 324 · cited 1× · see citing works
“In aviation, for example, research on ‘‘automation bias’’ has documented situations in which a human operator makes more errors when being assisted by a computerized device than when performing the same task without computer assistance.(5) Here we introduce an analysis method, applied to a case study in computer-aided…”
MethodGur, D., Sumkin, J. H., Rockette, H. E., Ganott, M. A., Hakim, C. M., Hardesty, L. A., Poller, W. R., Shah, R., & Wallace, L. P. (2004) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djh067 · Cited by 306 · see citing works
Single Reading with Computer-Aided Detection for Screening Mammography
Gilbert, F. J., Astley, S., Gillan, M. G., Agbaje, O. F., Wallis, M., James, J. J., Boggis, C. R. M., & Duffy, S. W. (2008) · New England Journal of Medicine · doi:10.1056/nejmoa0803545 · Cited by 296 · see citing works
On estimating binomial response relations
Anscombe, F. J. (1956) · Biometrika · doi:10.1093/biomet/43.3-4.461 · Cited by 284 · see citing works
Conceptual modeling of coincident failures in multiversion software
Littlewood, B., & Miller, D. R. (1989) · IEEE Transactions on Software Engineering · doi:10.1109/32.58771 · Cited by 251 · see citing works
A Goodness-of-Fit Test for Binary Regression Models, Based on Smoothing Methods
le Cessie, S., & van Houwelingen, H. C. (1991) · Biometrics · doi:10.2307/2532385 · Cited by 221 · see citing works
Taylor, P., & POTTS, H. W. W. (2008) · European Journal of Cancer · doi:10.1016/j.ejca.2008.02.016 · Cited by 160 · see citing works
Modeling software design diversity
Littlewood, B., Popov, P. T., & Strigini, L. (2001) · ACM Computing Surveys · doi:10.1145/384192.384195 · Cited by 147 · see citing works
Self-citationEffects of incorrect computer-aided detection (CAD) output on human decision-making in mammography
Alberdi, E., Povyakalo, A., Strigini, L., & Ayton, P. (2004) · Academic Radiology · doi:10.1016/j.acra.2004.05.012 · Cited by 124 · see citing works
Self-citationComputer-aided detection in mammography
Astley, S., & Gilbert, F. J. (2004) · Clinical Radiology · doi:10.1016/j.crad.2003.11.017 · Cited by 100 · see citing works
Zheng, B., Ganott, M. A., Britton, C. A., Hakim, C. M., Hardesty, L. A., Chang, T. S., Rockette, H. E., & Gur, D. (2001) · Radiology · doi:10.1148/radiol.2213010308 · Cited by 97 · see citing works
Clinical decision support: progress and opportunities
Lyman, J. A., Cohn, W. F., Bloomrosen, M., & Detmer, D. E. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.005561 · Cited by 63 · see citing works
Taylor, P., Champness, J., Given-Wilson, R., et al. (2005) · Health Technology Assessment · doi:10.3310/hta9060 · Cited by 63 · see citing works
Noble, M., Bruening, W., Uhl, S., & Schoelles, K. M. (2008) · Archives of Gynecology and Obstetrics · doi:10.1007/s00404-008-0841-y · Cited by 52 · see citing works
Alberdi, E., Povyakalo, A. A., Strigini, L., Ayton, P., Hartswood, M., Procter, R., & Slack, R. (2005) · British Journal of Radiology · doi:10.1259/bjr/37646417 · Cited by 48 · see citing works
Self-citationWagner, R. F., Beam, C. A., & Beiden, S. V. (2004) · Medical Decision Making · doi:10.1177/0272989x04271043 · Cited by 41 · see citing works
Taylor, P., Champness, J., Given-Wilson, R., et al. (2004) · British Journal of Radiology · doi:10.1259/bjr/34203805 · Cited by 36 · see citing works
Why Are People’s Decisions Sometimes Worse with Computer Support?
Alberdi, E., Strigini, L., Povyakalo, A. A., & Ayton, P. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-04468-7_3 · Cited by 29 · see citing works
Self-citationAssessing the impact of CAD on the sensitivity and specificity of film readers
Taylor, P. M., Given-Wilson, R., Champness, J., POTTS, H. W. W., & Johnston, K. (2004) · Clinical Radiology · doi:10.1016/j.crad.2004.04.017 · Cited by 22 · see citing works
Hosmer (1997) · doi:10.1002/(sici)1097-0258(19970515)16:9
Flagged first, then by citations
Research works printed but not linked: 5
- [13]13. Strigini L, Povyakalo AA, Alberdi E. Human-machine diversity in the use of computerised advisory systems: a case study. Int Conf on Dependable Systems and Networks (DSN'03) June 22 -25 2003; San Francisco: IEEE.
- [14]14. Littlewood B, Miller DR. Conceptual modelling of coincident failures in multi- version software. IEEE Trans Software Engineering. 1989;15(12):1596-614.
- [15]15. Littlewood B, Popov P, Strigini L. Modelling software design diversity -a review. ACM Computing Surveys. 2002;33(2):177-208.
- [16]16. Alberdi E, Povyakalo AA, Strigini L, Ayton P. Effects of incorrect CAD output on human decision making in mammography. Acad Radiol. 2004;11(8):909-18.
- [29]29. Kirkwood BR, Sterne JAC. Essentials of medical statistics. Second ed. Oxford: Blackwell Science; 2003.
Cited by
Sorted by citations
Unintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066 · see citing works
Do as AI say: susceptibility in deployment of clinical decision-aids
Gaube, S., Suresh, H., Raue, M., Merritt, A., Berkowitz, S. J., Lermer, E., Coughlin, J. F., Guttag, J. V., Colak, E., & Ghassemi, M. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00385-9 · Cited by 543 · see citing works
The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441 · see citing works
Automation bias and verification complexity: a systematic review
Lyell, D., & Coiera, E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw105 · Cited by 451 · see citing works
Abdelwanis, M., Alarafati, H. K., Tammam, M. M. S., & Simsekler, M. C. E. (2024) · Journal of Safety Science and Resilience · doi:10.1016/j.jnlssr.2024.06.001 · Cited by 137 · see citing works
Automation bias in electronic prescribing
Lyell, D., Magrabi, F., Raban, M. Z., Pont, L. G., Baysari, M. T., Day, R. O., & Coiera, E. (2017) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-017-0425-5 · Cited by 103 · see citing works
“There have been a small number of studies conducted in healthcare, finding evidence of AB omission errors in computer-aided detection of cancers in mammograms, [15, 16] and”
BackgroundLevy, A., Agrawal, M., Satyanarayan, A., & Sontag, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445522 · Cited by 73 · see citing works
Horowitz, M. C., & Kahn, L. (2024) · International Studies Quarterly · doi:10.1093/isq/sqae020 · Cited by 68 · see citing works
Lekadir, K., Osuala, R., Gallin, C., Lazrak, N., Kushibar, K., Tsakou, G., Aussó, S., Alberich, L. C., Marias, K., Tsiknakis, M., et al. (2021) · arXiv · doi:10.48550/arxiv.2109.09658 · Cited by 58 · see citing works
Misplaced Trust: Measuring the Interference of Machine Learning in Human Decision-Making
Suresh, H., Lao, N., & Liccardi, I. (2020) · 12th ACM Conference on Web Science · doi:10.1145/3394231.3397922 · Cited by 49 · see citing works
- Authors
- Povyakalo, A. A., Alberdi, E., Strigini, L., & Ayton, P.
- Affiliations
- City, University of London (GB)
- Venue
- Medical Decision Making
- Publisher
- SAGE Publications
- Published
- 2013-01
- Volume
- 33
- Issue
- 1
- Pages
- 98-107
- ISSN
- 0272-989X (ISSN-L)
- 1552-681X
- OpenAlex
- W1966744297
- Cited by
- 79
- References
- 30
- Citations by year
- 2026: 4 · 2025: 12 · 2024: 7 · 2023: 6 · 2022: 11 · 2021: 13 · 2020: 9 · 2019: 4 · 2018: 2 · 2017: 5 · 2016: 1 · 2015: 2 · 2014: 2 · 2013: 1
- FWCI
- 2.21
- Type
- journal article
- Language
- en
- Open access
- Locations
- Medical Decision Making (publishedVersion)
- PubMed (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- City Research Online (City University London) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- University of Pennsylvania
- Cancer Research UK (GR/C22515/A7339)
- University College London
- Engineering and Physical Sciences Research Council (EP/E000517/1)
- Health Technology Assessment Programme
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
- Health Sciences › Medicine › Pulmonary and Respiratory Medicine › Digital Radiography and Breast Imaging
- Keywords
- computer-aided detection, breast cancer screening, mammography, diagnostic sensitivity, case difficulty, decision quality assessment, alert system
- MeSH terms
- Breast Neoplasms, Diagnosis, Computer-Assisted, Female, Humans, Mammography, Mass Screening, Probability, Sensitivity and Specificity, Logistic Models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
