Studying human-AI collaboration protocols: the case of the Kasparov’s law in radiological double reading
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Purpose: The integration of Artificial Intelligence into medical practices has recently been advocated for the promise to bring increased efficiency and effectiveness to these practices. Nonetheless, little research has so far been aimed at understanding the best human-AI interaction protocols in collaborative tasks, even in currently more viable settings, like independent double-reading screening tasks. Methods: To this aim, we report about a retrospective case–control study, involving 12 board-certified radiologists, in the detection of knee lesions by means of Magnetic Resonance Imaging, in which we simulated the serial combination of two Deep Learning models with humans in eight double-reading protocols. Inspired by the so-called Kasparov’s Laws, we investigate whether the combination of humans and AI models could achieve better performance than AI models alone, and whether weak reader, when supported by fit-for-use interaction protocols, could out-perform stronger readers. Results: We discuss two main findings: groups of humans who perform significantly worse than a state-of-the-art AI can significantly outperform it if their judgements are aggregated by majority voting (in concordance with the first part of the Kasparov’s law); small ensembles of significantly weaker readers can significantly outperform teams of stronger readers, supported by the same computational tool, when the judgments of the former ones are combined within “fit-for-use” protocols (in concordance with the second part of the Kasparov’s law). Conclusion: Our study shows that good interaction protocols can guarantee improved decision performance that easily surpasses the performance of individual agents, even of realistic super-human AI systems. This finding highlights the importance of focusing on how to guarantee better co-operation within human-AI teams, so to enable safer and more human sustainable care practices.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Nagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020) · BMJ · doi:10.1136/bmj.m689 · Cited by 1,152
Unintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066
Self-citationInteractive machine learning for health informatics: when do we need the human-in-the-loop?
Holzinger, A. (2016) · Brain Informatics · doi:10.1007/s40708-016-0042-6 · Cited by 906
Machines as teammates: A research agenda on AI in team collaboration
Seeber, I., Bittner, E., Briggs, R. O., de Vreede, T., de Vreede, G.-J., Elkins, A., Maier, R., Merz, A. B., Oeste-Reiß, S., Randrup, N., et al. (2020) · Information & Management · doi:10.1016/j.im.2019.103174 · Cited by 835
Coordination mechanisms: Towards a conceptual foundation of CSCW systems design
Schmidt, K., & Simone, C. (1996) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/bf00133655 · Cited by 822
Diagnostic Accuracy of Digital Screening Mammography With and Without Computer-Aided Detection
Lehman, C. D., Wellman, R. D., Buist, D. S. M., Kerlikowske, K., Tosteson, A. N. A., & Miglioretti, D. L. (2015) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2015.5231 · Cited by 678
Potential liability for physicians using artificial intelligence
Price, W. N., Gerke, S., & Cohen, I. G. (2019) · JAMA · doi:10.1001/jama.2019.15064 · Cited by 579
Automation bias and verification complexity: a systematic review
Lyell, D., & Coiera, E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw105 · Cited by 451
Akata, Z., Balliet, D., de Rijke, M., Dignum, F., Dignum, V., Eiben, G., Fokkens, A., Grossi, D., Hindriks, K., Hoos, H., et al. (2020) · Computer · doi:10.1109/mc.2020.2996587 · Cited by 415
Deep learning enables reduced gadolinium dose for contrast-enhanced brain MRI
Gong, E., Pauly, J. M., Wintermark, M., & Zaharchuk, G. (2018) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.25970 · Cited by 351
A solution to the single-question crowd wisdom problem
Prelec, D., Seung, H. S., & McCoy, J. (2017) · Nature · doi:10.1038/nature21054 · Cited by 314
Are Current Tort Liability Doctrines Adequate for Addressing Injury Caused by AI?
(2019) · AMA Journal of Ethics · doi:10.1001/amajethics.2019.160 · Cited by 188
Barnett, M. L., Boddupalli, D., Nundy, S., & Bates, D. W. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.0096 · Cited by 154
Human Factors Challenges for the Safe Use of Artificial Intelligence in Patient Care
Sujan, M., Furniss, D., Grundy, K., Grundy, H., Nelson, D., Elliott, M., White, S., Habli, I., & Reynolds, N. (2019) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2019-100081 · Cited by 144
Automation Bias and Errors: Are Crews Better Than Individuals?
Skitka, L. J., Mosier, K. L., Burdick, M., & Rosenblatt, B. (2000) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap1001_5 · Cited by 127
Effects of incorrect computer-aided detection (CAD) output on human decision-making in mammography
Alberdi, E., Povyakalo, A., Strigini, L., & Ayton, P. (2004) · Academic Radiology · doi:10.1016/j.acra.2004.05.012 · Cited by 124
Added value of double reading in diagnostic radiology,a systematic review
Geijer, H., & Geijer, M. (2018) · Insights into Imaging · doi:10.1007/s13244-018-0599-0 · Cited by 117
AI outperforms radiologists in mammographic screening
Killock, D. (2020) · Nature Reviews Clinical Oncology · doi:10.1038/s41571-020-0329-7 · Cited by 86
Are we at risk of groupthink in our approach to teamwork interventions in health care?
Kaba, A., Wishart, I., Fraser, K., Coderre, S., & McLaughlin, K. (2016) · Medical Education · doi:10.1111/medu.12943 · Cited by 83
UK radiologist staffing crisis reaches critical levels
Gourd, E. (2017) · The Lancet Oncology · doi:10.1016/s1470-2045(17)30806-9 · Cited by 12
Cited by
Natali, C., Marconi, L., Dias Duran, L. D., & Cabitza, F. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11352-1 · Cited by 143
“For example, by tracking clinician–AI interactions and decision outcomes in real time, subtle shifts-such as delays in clinician response Vallor ( 2015 ), Green ( 2019 ), Dias Duran ( 2021 ), Cabitza ( 2021 ), Rafner et al. ( 2022 ), Winter and Carusi ( 2022a ), Wessel ( 2023 )”
InfluentialSelf-citationRams, hounds and white boxes: investigating human–AI collaboration protocols in medical diagnosis
Cabitza, F., Campagner, A., Ronzio, L., Cameli, M., Mandoli, G. E., Pastore, M. C., Sconfienza, L. M., Folgado, D., Barandas, M., & Gamboa, H. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102506 · Cited by 117
Self-citationGodwin, R. C., Bryant, A. S., Wagener, B. M., Ness, T. J., DeBerry, J. J., Horn, L. L., Graves, S. H., Archer, A. C., & Melvin, R. L. (2024) · SoftwareX · doi:10.1016/j.softx.2023.101601 · Cited by 15
From testbeds to high-stakes work: a review of human-AI teaming domains and teaming factors
Kargarnovin, S., Hernandez, C. I., Reiners, D., Cruz-Neira, C., Bochenek, G., & Karwowski, W. (2026) · Frontiers in Robotics and AI · doi:10.3389/frobt.2026.1733942 · Cited by 2
“This supports the broader claim that performance in human-AI teams is often a property of the interaction protocol, how judgements are combined, when AI is consulted, and how disagreement is resolved, rather than a simple reflection of the AI model’s standalone accuracy (Cabitza et al., 2021).”
Metadata
- Authors
- Cabitza, F., Campagner, A., & Sconfienza, L. M.
- Venue
- Health Information Science and Systems
- Published
- 2021-02-05
- Volume
- 9
- Issue
- 1
- Pages
- 8-8
- ISSN
- 2047-2501 (ISSN-L)
- OpenAlex
- W3131191101
- Cited by
- 39
- References
- 20
- FWCI
- 1.08
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.