Comparative Accuracy of Diagnosis by Collective Intelligence of Multiple Physicians vs Individual Physicians
Open accessAbstract
Importance: The traditional approach of diagnosis by individual physicians has a high rate of misdiagnosis. Pooling multiple physicians' diagnoses (collective intelligence) is a promising approach to reducing misdiagnoses, but its accuracy in clinical cases is unknown to date. Objective: To assess how the diagnostic accuracy of groups of physicians and trainees compares with the diagnostic accuracy of individual physicians. Design, Setting, and Participants: Cross-sectional study using data from the Human Diagnosis Project (Human Dx), a multicountry data set of ranked differential diagnoses by individual physicians, graduate trainees, and medical students (users) solving user-submitted, structured clinical cases. From May 7, 2014, to October 5, 2016, groups of 2 to 9 randomly selected physicians solved individual cases. Data analysis was performed from March 16, 2017, to July 30, 2018. Main Outcomes and Measures: The primary outcome was diagnostic accuracy, assessed as a correct diagnosis in the top 3 ranked diagnoses for an individual; for groups, the top 3 diagnoses were a collective differential generated using a weighted combination of user diagnoses with a variety of approaches. A version of the McNemar test was used to account for clustering across repeated solvers to compare diagnostic accuracy. Results: Of the 2069 users solving 1572 cases from the Human Dx data set, 1228 (59.4%) were residents or fellows, 431 (20.8%) were attending physicians, and 410 (19.8%) were medical students. Collective intelligence was associated with increasing diagnostic accuracy, from 62.5% (95% CI, 60.1%-64.9%) for individual physicians up to 85.6% (95% CI, 83.9%-87.4%) for groups of 9 (23.0% difference; 95% CI, 14.9%-31.2%; P < .001). The range of improvement varied by the specifications used for combining groups' diagnoses, but groups consistently outperformed individuals regardless of approach. Absolute improvement in accuracy from individuals to groups of 9 varied by presenting symptom from an increase of 17.3% (95% CI, 6.4%-28.2%; P = .002) for abdominal pain to 29.8% (95% CI, 3.7%-55.8%; P = .02) for fever. Groups from 2 users (77.7% accuracy; 95% CI, 70.1%-84.6%) to 9 users (85.5% accuracy; 95% CI, 75.1%-95.9%) outperformed individual specialists in their subspecialty (66.3% accuracy; 95% CI, 59.1%-73.5%; P < .001 vs groups of 2 and 9). Conclusions and Relevance: A collective intelligence approach was associated with higher diagnostic accuracy compared with individuals, including individual specialists whose expertise matched the case diagnosis, across a range of medical cases. Given the few proven strategies to address misdiagnosis, this technique merits further study in clinical settings.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Expert disagreement is prevalent in medical decision-making tasks.
“However, clinicians can be biased in their decision making and expert disagreement is prevalent in medical decision-making tasks [7, 33, 34].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [7]
The Discussion reports that final diagnoses differed from referral diagnoses in 21% of subspecialist referrals, supporting disagreement in a specific diagnostic setting. It does not establish that expert disagreement is prevalent across medical decision-making tasks generally; the paper also notes that most referral diagnoses were unchanged or only refined. (LLM, conf=0.82)
Experimental — accuracy not yet measuredAggregating informed opinions improves effectiveness, promoting the wisdom of crowds.
“extensive evidence has shown that aggregating informed opinions improves effectiveness [50 , 58] , promoting the wisdom of crowds [59 , 60] .”
From Enhancing human-AI collaboration: The case of colonoscopy (2024) · cites this paper as [58]
Clinicians can be biased in their decision making.
“However, clinicians can be biased in their decision making and expert disagreement is prevalent in medical decision-making tasks [7, 33, 34].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [7]
The Introduction discusses misdiagnosis, but does not attribute it to bias in clinicians’ decision-making. The Discussion > Limitations mentions possible bias in the study’s assessment of diagnostic solutions, not bias in clinicians’ decisions. Searches for cognitive bias and biased physician judgments found no passage addressing the attributed claim. (LLM, conf=0.75)
Experimental — accuracy not yet measuredReference diagnoses were obtained using a previously described collective intelligence approach, which results in more reproducible diagnoses than diagnoses obtained by individual dermatologist review.
“Reference diagnoses were obtained using a previously described collective intelligence approach, which results in more reproducible diagnoses than diagnoses obtained by individual dermatologist review (eTable 2 in the Supplement).26,30”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Fitting Linear Mixed-Effects Models Using lme4Preprint
Bates, D. M., Maechler, M., Bolker, B. M., & Walker, S. (2014) · arXiv (Cornell University) · doi:10.3929/ethz-b-000105397 · Cited by 9,435 · see citing works
Self-citationthe delphi method: techniques and applications
Linstone, H. A., & Turoff, M. (1975) · Bibliothèque et Archives nationales du Québec (Québec government) · Cited by 5,236 · see citing works
Qualitative Research: Consensus methods for medical and health services research
Jones, J., & Hunter, D. (1995) · BMJ · doi:10.1136/bmj.311.7001.376 · Cited by 3,471 · see citing works
Evidence for a Collective Intelligence Factor in the Performance of Human Groups
Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N., & Malone, T. W. (2010) · Science · doi:10.1126/science.1193147 · Cited by 2,685 · see citing works
The Delphi Method: Techniques and Applications
J., J. E., Linstone, H. A., & Turoff, M. (1976) · Technometrics · doi:10.2307/1268751 · Cited by 2,245 · see citing works
Improving Diagnosis in Health Care
Committee on Diagnostic Error in Health Care, Board on Health Care Services, Institute of Medicine, & The National Academies of Sciences, E. (2015) · doi:10.17226/21794 · Cited by 1,376 · see citing works
Singh, H., Meyer, A. N. D., & Thomas, E. J. (2014) · BMJ Quality & Safety · doi:10.1136/bmjqs-2013-002627 · Cited by 677 · see citing works
The incidence of diagnostic error in medicine
Graber, M. L. (2013) · BMJ Quality & Safety · doi:10.1136/bmjqs-2012-001615 · Cited by 626 · see citing works
(2008) · doi:10.1126/science.1157679 · Cited by 406 · see citing works
The Global Burden of Diagnostic Errors in Primary Care
Singh, H., Schiff, G. D., Graber, M. L., Onakpoya, I., & Thompson, M. J. (2017) · BMJ Quality & Safety · doi:10.1136/bmjqs-2016-005401 · Cited by 395 · see citing works
Decker, S. L. (2012) · Health Affairs · doi:10.1377/hlthaff.2012.0294 · Cited by 360 · see citing works
Beam, C. A. (1996) · Archives of Internal Medicine · doi:10.1001/archinte.156.2.209 · Cited by 337 · see citing works
Access To Specialty Care And Medical Services In Community Health Centers
Cook, N. L., Hicks, L. S., O’Malley, A. J., Keegan, T., Guadagnoli, E., & Landon, B. E. (2007) · Health Affairs · doi:10.1377/hlthaff.26.5.1459 · Cited by 280 · see citing works
Physicians’ Diagnostic Accuracy, Confidence, and Resource Requests: A Vignette Study
Meyer, A. N. D., Payne, V. L., Meeks, D. W., Rao, R., & Singh, H. (2013) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2013.10081 · Cited by 268 · see citing works
Variability in the Interpretation of Screening Mammograms by US Radiologists
Beam, C. A. (1996) · Archives of Internal Medicine · doi:10.1001/archinte.1996.00440020119016 · Cited by 263 · see citing works
When are two heads better than one and why?
Koriat, A. (2012) · Science · doi:10.1126/science.1216549 · Cited by 256 · see citing works
Assessing the Accuracy of Administrative Data in Health Information Systems
Peabody, J. W., Luck, J., Jain, S., Bertenthal, D., & Glassman, P. A. (2004) · Medical Care · doi:10.1097/00005650-200411000-00005 · Cited by 254 · see citing works
Diagnostic errors in three medical eras: a necropsy study
Sonderegger-lseli, K., Burger, S., Muntwyler, J., & Salomon, F. (2000) · The Lancet · doi:10.1016/s0140-6736(00)02349-7 · Cited by 215 · see citing works
Boosting medical diagnostics by pooling independent judgments
Kurvers, R. H. J. M., Herzog, S. M., Hertwig, R., Krause, J., Carney, P. A., Bogart, A., Argenziano, G., Zalaudek, I., & Wolf, M. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1601827113 · Cited by 198 · see citing works
Wolf, M., Krause, J., Carney, P. A., Bogart, A., & Kurvers, R. H. J. M. (2015) · PLoS ONE · doi:10.1371/journal.pone.0134269 · Cited by 157 · see citing works
Diagnostic Performance by Medical Students Working Individually or in Teams
Hautz, W. E., Kämmer, J. E., Schauber, S. K., Spies, C., & Gaissmaier, W. (2015) · JAMA · doi:10.1001/jama.2014.15770 · Cited by 90 · see citing works
Kämmer, J. E., Hautz, W. E., Herzog, S. M., Kunina‐Habenicht, O., & Kurvers, R. H. J. M. (2017) · Medical Decision Making · doi:10.1177/0272989x17696998 · Cited by 79 · see citing works
Extent of diagnostic agreement among medical referrals
Van Such, M., Lohr, R., Beckman, T., & Naessens, J. M. (2017) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12747 · Cited by 75 · see citing works
Detection Accuracy of Collective Intelligence Assessments for Skin Cancer Diagnosis
Kurvers, R. H. J. M., Krause, J., Argenziano, G., Zalaudek, I., & Wolf, M. (2015) · JAMA Dermatology · doi:10.1001/jamadermatol.2015.3149 · Cited by 64 · see citing works
Autopsy as an Outcome and Performance Measure: Summary
Shojania, K., Ec, B., McDonald, K. M., et al. (2002) · Cited by 18 · see citing works
Fitting Linear Mixed-Effects Models using lme4
Bates, D., Mächler, M., Bolker, B., et al. (2014) · arXiv · doi:10.48550/arxiv.1406.5823
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- Surowiecki, 2005, The Wisdom of Crowds
Cited by
Sorted by citations
A deep learning system for differential diagnosis of skin diseases
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0842-3 · Cited by 853 · see citing works
Superior skin cancer classification by the combination of human and artificial intelligence
Hekler, A., Utikal, J. S., Enk, A. H., Hauschild, A., Weichenthal, M., Maron, R. C., Berking, C., Haferkamp, S., Klode, J., Schadendorf, D., et al. (2019) · European Journal of Cancer · doi:10.1016/j.ejca.2019.07.019 · Cited by 357 · see citing works
Hirosawa, T., Harada, Y., Yokose, M., Sakamoto, T., Kawamura, R., & Shimizu, T. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20043378 · Cited by 344 · see citing works
“Additionally, consistent differential diagnoses made by two independent physicians were suggested to be important differential diagnoses, which were collective intelligence to reduce misdiagnoses [32].”
BackgroundMajkowska, A., Mittal, S., Steiner, D. F., Reicher, J. J., McKinney, S. M., Duggan, G. E., Eswaran, K., Cameron Chen, P.-H., Liu, Y., Kalidindi, S. R., et al. (2020) · Radiology · doi:10.1148/radiol.2019191293 · Cited by 277 · see citing works
“…of adjudicated labels, extensive work has been performed on the value of multiple interpretations in mammography (30) and other clinical settings (31,32). the most obviously positive and negative for pneumothorax, opacity, nodule or mass, or fracture for training, which can fail to address the…”
BackgroundAmbiguous Medical Image Segmentation Using Diffusion Models
Rahman, A., Valanarasu, J. M. J., Hacihaliloglu, I., & Patel, V. M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01110 · Cited by 227 · see citing works
Jain, A., Way, D., Gupta, V., Gao, Y., de Oliveira Marinho, G., Hartford, J., Sayres, R., Kanada, K., Eng, C., Nagpal, K., et al. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.7249 · Cited by 164 · see citing works
“of patients included in study 1016 152 Age, median (IQR), ya 43 (30-56) 49 (35-59) Sex, No.”
A Human-AI Collaborative Approach for Clinical Decision Making on Rehabilitation Assessment
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445472 · Cited by 137 · see citing works
“In most medical disciplines, experts can be biased in their decision making based on their own experiences and expert disagreement is prevalent [5, 30, 31], even if they rely on standardized guidelines [18, 23, 53].”
BackgroundUchino, E., Suzuki, K., Sato, N., Kojima, R., Tamada, Y., Hiragi, S., Yokoi, H., Yugami, N., Minamiguchi, S., Haga, H., et al. (2020) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2020.104231 · Cited by 93 · see citing works
Girdhar, N., Sinha, A., & Gupta, S. (2022) · Soft Computing · doi:10.1007/s00500-022-07406-z · Cited by 85 · see citing works
“This is further worsened by the expected number of deaths this year, that estimates to about 6,06,520 (Barnett et al. 2019).”
BackgroundRetractedLee, M. H., & Chew, C. J. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610218 · Cited by 78 · see citing works
“However, clinicians can be biased in their decision making and expert disagreement is prevalent in medical decision-making tasks [7, 33, 34].”
BackgroundThe network science of collective intelligence
Centola, D. M. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.08.009 · Cited by 78 · see citing works
“The study of collective intelligence impacts a remarkable range of disciplines, including fi nance [12], national elections [13,14], sports betting [15], medical decision-making [16 – 18], marketing [19], engineering and data sciences [5,20,21], and animal behavior [22 – 24], to name just a few.”
BackgroundDiagnostic error: incidence, impacts, causes and preventive strategies
Scott, I. A., & Crock, C. (2020) · The Medical Journal of Australia · doi:10.5694/mja2.50771 · Cited by 73 · see citing works
Collective intelligence in medical decision-making: a systematic scoping review
Radcliffe, K. G., Lyson, H. C., Barr‐Walker, J., & Sarkar, U. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0882-0 · Cited by 63 · see citing works
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415227 · Cited by 57 · see citing works
“As experts often rely on their experience-based heuristics, disagreement on decision making is prevalent [4, 30, 31].”
BackgroundThe cases for and against artificial intelligence in the medical school curriculum
Ngo, B., Nguyen, D., & vanSonnenberg, E. (2022) · Radiology: Artificial Intelligence · doi:10.1148/ryai.220074 · Cited by 53 · see citing works
Rohaut, B., Calligaris, C., Hermann, B., Perez, P., Faugeras, F., Raimondo, F., King, J. R., Engemann, D., Marois, C., Le Guennec, L., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03019-1 · Cited by 62 · see citing works
“Finally, to optimize group decision-making and mitigate potential biases, we adhere to basic principles 13,30,31 .”
Ambiguity-aware AI Assistants for Medical Data Analysis
Schaekermann, M., Beaton, G., Sanoubari, E., Lim, A. S. P., Larson, K. S., & Law, E. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376506 · Cited by 54 · see citing works
“Despite the abundance of standardized medical guidelines [4], expert disagreement is prevalent across medical disciplines [5, 54], making our approach useful beyond the specific domain of sleep health.”
BackgroundUnderstanding Expert Disagreement in Medical Data Analysis through Structured Adjudication
Schaekermann, M., Beaton, G., Habib, M., Lim, A. S. P., Larson, K. S., & Law, E. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359178 · Cited by 52 · see citing works
“[7] showed that automatic pooling of independent opinions from multiple doctors outperformed individual diagnosis across various diagnostic tasks.”
BackgroundHuman–AI collectives most accurately diagnose clinical vignettes
Zöller, N., Berger, J., Lin, I., Fu, N., Komarneni, J., Barabucci, G., Laskowski, K., Shia, V., Harack, B., Chu, E. A., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2426153122 · Cited by 62 · see citing works
Raffel, K. E., Kantor, M. A., Barish, P. N., Esmaili, A. M., Lim, H., Xue, F., & Ranji, S. R. (2020) · BMJ Quality & Safety · doi:10.1136/bmjqs-2020-010896 · Cited by 45 · see citing works
Barwise, A., Leppin, A., Dong, Y., Huang, C., Pinevich, Y., Herasevich, S., Soleimani, J., Gajic, O., Pickering, B., & Kumbamu, A. (2021) · Journal of Patient Safety · doi:10.1097/pts.0000000000000817 · Cited by 43 · see citing works
“A qualitative study by Thomas and Newman-Toker did highlight that barriers to diagnosis included sociocultural and institutional norms that suggested that making a diagnosis was the purview of physicians; however, a well-functioning interdisciplinary team can facilitate a timely and accurate diagnosis and should be le…”
BackgroundCentola, D. M., Guilbeault, D., Sarkar, U., Khoong, E. C., & Zhang, J. (2021) · Nature Communications · doi:10.1038/s41467-021-26905-5 · Cited by 43 · see citing works
Teamwork in clinical reasoning – cooperative or parallel play?
Olson, A. P. J., Durning, S. J., Fernandez Branson, C., Sick, B., Lane, K. P., & Rencic, J. J. (2020) · Diagnosis · doi:10.1515/dx-2020-0020 · Cited by 43 · see citing works
Cabitza, F., Campagner, A., & Sconfienza, L. M. (2021) · Health Information Science and Systems · doi:10.1007/s13755-021-00138-8 · Cited by 44 · see citing works
Imaging modalities for inguinal hernia diagnosis: a systematic review
Piga, E., Zetner, D. B., Andresen, K., & Rosenberg, J. (2020) · Hernia · doi:10.1007/s10029-020-02189-4 · Cited by 39 · see citing works
“Studies using multiple examiners per patient must be expected to perform better due to possible additive effects [49].”
BackgroundTsang, M. B., Schwalm, J.-D., Gandhi, S., Sibbald, M., Gafni, A., Mercuri, M., Salehian, O., Lamy, A. L., Pericak, D., Jolly, S. S., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.12749 · Cited by 36 · see citing works
Chung, M., Moon, W.-K., & Jones-Jang, S. M. (2023) · Digital Journalism · doi:10.1080/21670811.2023.2254820 · Cited by 41 · see citing works
“This finding reflects the dilemma of crowdsourcing; crowd-sourced fact-checking may be perceived to be impartial because the information has been balanced by a large number of people with varying ideas, beliefs, and knowledge (Barnett et al. 2019).”
Provider Misinterpretation, Documentation, and Follow-Up of Definitive Urine Drug Testing Results
Chua, I. S., Petrides, A. K., Schiff, G. D., Ransohoff, J. R., Kantartjis, M., Streid, J. L., Demetriou, C. A., & Melanson, S. E. F. (2019) · Journal of General Internal Medicine · doi:10.1007/s11606-019-05514-5 · Cited by 32 · see citing works
“There is evidence to suggest a collective intelligence approach is associated with higher diagnostic accuracy.(48) One study implemented a pharmacist econsult UDT interpretation service that could help guide provider follow-up actions.”
BackgroundAutomating hybrid collective intelligence in open-ended medical diagnostics
Kurvers, R. H. J. M., Nuzzolese, A. G., Russo, A., Barabucci, G., Herzog, S. M., & Trianni, V. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2221473120 · Cited by 32 · see citing works
“(32) used data from a large medical crowdsourcing platform, the Human Diagnosis Project [Human Dx, https://www.”
Methodde Jong, J. S. Y., Blok, M. R. S., Thijs, R. D., Harms, M. P. M., Hemels, M. E. W., de Groot, J. R., van Dijk, N., & de Lange, F. J. (2020) · EP Europace · doi:10.1093/europace/euaa345 · Cited by 31 · see citing works
“In addition, a multidisciplinary panel evaluated patients after long-term follow-up in order to determine the diagnostic accuracy and safety of the diagnosis established in the syncope unit.(2,7,8) Methods”
BackgroundEnhancing human-AI collaboration: The case of colonoscopy
Introzzi, L., Zonca, J., Cabitza, F., Cherubini, P., & Reverberi, C. (2024) · Digestive and Liver Disease · doi:10.1016/j.dld.2023.10.018 · Cited by 28 · see citing works
Developing critical thinking skills for delivering optimal care
Scott, I. A., Hubbard, R. E., Crock, C., Campbell, T. G., & Perera, M. (2021) · Internal Medicine Journal · doi:10.1111/imj.15272 · Cited by 28 · see citing works
Expert Discussions Improve Comprehension of Difficult Cases in Medical Image Assessment
Schaekermann, M., Cai, C. J., Huang, A. E., & Sayres, R. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376290 · Cited by 28 · see citing works
“Several works have suggested ways to make productive use of disagreement information in medical data labels (e.g., [2, 3, 8, 19, 28, 33, 36]).”
BackgroundWinkler, J. K., Sies, K., Fink, C., Toberer, F., Enk, A., Abassi, M. S., Fuchs, T., Blum, A., Stolz, W., Coras‐Stepanek, B., et al. (2021) · JDDG Journal der Deutschen Dermatologischen Gesellschaft · doi:10.1111/ddg.14510 · Cited by 26 · see citing works
Belief rule mining using the evidential reasoning rule for medical diagnosis
Chang, L., Fu, C., Zhu, W., & Liu, W. (2020) · International Journal of Approximate Reasoning · doi:10.1016/j.ijar.2020.12.009 · Cited by 23 · see citing works
Combining Multiple Large Language Models Improves Diagnostic Accuracy
Barabucci, G., Shia, V., Chu, E., Harack, B., Laskowski, K., & Fu, N. (2024) · NEJM AI · doi:10.1056/aics2400502 · Cited by 24 · see citing works
“The degree of increase in accuracy achieved by the method employed in this study is in line with similar results in the field of collective intelligence, both in the medical field [11, 12] and outside [10, 13].”
BackgroundMethodResultInfluentialChow, R. A., Tan, N., Henry, T. S., et al. (2020) · Academic Radiology · doi:10.1016/j.acra.2020.01.015 · Cited by 21 · see citing works
Remote Tool-Based Adjudication for Grading Diabetic Retinopathy
Schaekermann, M., Hammel, N., Terry, M., Ali, T. K., Liu, Y., Basham, B., Campana, B., Chen, W., Ji, X., Krause, J., et al. (2019) · Translational Vision Science & Technology · doi:10.1167/tvst.8.6.40 · Cited by 19 · see citing works
Krockow, E. M., Kurvers, R. H. J. M., Herzog, S. M., Kämmer, J. E., Hamilton, R. A., Thilly, N., Macheda, G., & Pulcini, C. (2020) · Scientific Reports · doi:10.1038/s41598-020-75063-z · Cited by 19 · see citing works
Clinical decision support in primary care for better diagnosis and management of retinal disease
Ho, S., Kalloniatis, M., & Ly, A. (2022) · Clinical and Experimental Optometry · doi:10.1080/08164622.2021.2008791 · Cited by 20 · see citing works
“They can also act as a source of second opinion, which has been shown to increase diagnostic accuracy by as much as a third.(12) Recently, artificial intelligence has come to the forefront of clinical decision support development, sparking tremendous interest due to its potential to increase accuracy and efficiency in…”
BackgroundNaccache, L., Luauté, J., Silva, S., Sitt, J. D., & Rohaut, B. (2021) · Revue Neurologique · doi:10.1016/j.neurol.2021.12.004 · Cited by 18 · see citing works
Retrieval-Based Diagnostic Decision Support: Mixed Methods Study
Abdullahi, T., Mercurio, L., Singh, R., & Eickhoff, C. (2024) · JMIR Medical Informatics · doi:10.2196/50209 · Cited by 20 · see citing works
“Studies [2,3] have shown that, rather than relying on a single physician for a final diagnosis, obtaining recommendations from multiple physicians increases diagnostic accuracy.”
BackgroundPsychiatric diagnostic dilemmas among people with intellectual and developmental disabilities
Charlot, L., Hodge, S. M., Holland, A., & Frazier, J. A. (2022) · Journal of Intellectual Disability Research · doi:10.1111/jir.12972 · Cited by 17 · see citing works
An equation for excellence in clinical reasoning
Shimizu, T., & Graber, M. L. (2022) · Diagnosis · doi:10.1515/dx-2022-0060 · Cited by 17 · see citing works
Collective Intelligence Increases Diagnostic Accuracy in a General Practice Setting
Blanchard, M. D., Herzog, S. M., Kämmer, J. E., Zöller, N., Kostopoulou, O., & Kurvers, R. H. J. M. (2024) · Medical Decision Making · doi:10.1177/0272989x241241001 · Cited by 15 · see citing works
“39 However, there is a paucity of research that has applied this approach to more ill-defined medical environments, 35,40,41 such as emergency medicine (i.e., high time pressure and incomplete information 42 ) or general practice.”
BackgroundRoot cause analysis of cases involving diagnosis
Graber, M. L., Castro, G. M., Danforth, M., Tilly, J., Croskerry, P., El-Kareh, R. E., Hemmalgarn, C., Ryan, R., Tozier, M., Trowbridge, B., et al. (2024) · Diagnosis · doi:10.1515/dx-2024-0102 · Cited by 18 · see citing works
Gender in the Markets for Expertise
de Vaan, M., & Stuart, T. E. (2022) · American Sociological Review · doi:10.1177/00031224221087374 · Cited by 32 · see citing works
“Some of the literature indicates SOs improve patient health outcomes because treatment decisions are “wiser” (Barnett et al. 2019).”
BackgroundBuilding Global Societies on Collective Intelligence: Challenges and Opportunities
Suran, S., Pattanaik, V., Kurvers, R. H. J. M., Hallin, C. A., Liddo, A. D., Krimmer, R., & Draheim, D. (2021) · Digital Government Research and Practice · doi:10.1145/3568169 · Cited by 15 · see citing works
Basic Artificial Intelligence Techniques
Kalpathy–Cramer, J., Patel, J., Bridge, C. P., & Chang, K. (2021) · Radiologic Clinics of North America · doi:10.1016/j.rcl.2021.06.005 · Cited by 14 · see citing works
Iyngkaran, P., Thomas, M. C., Neil, C., Jelinek, M. V., Cooper, M. E., Horowitz, J. D., Hare, D. L., & Kaye, D. M. (2020) · Current Heart Failure Reports · doi:10.1007/s11897-020-00454-2 · Cited by 14 · see citing works
Handshake antimicrobial stewardship as a model to recognize and prevent diagnostic errors
Searns, J. B., Williams, M. C., MacBrayne, C. E., Wirtz, A. L., Leonard, J. E., Boguniewicz, J., Parker, S. K., & Grubenhoff, J. A. (2020) · Diagnosis · doi:10.1515/dx-2020-0032 · Cited by 14 · see citing works
Uchino, E., Suzuki, K., Sato, N., Kojima, R., Tamada, Y., Hiragi, S., Yokoi, H., Yugami, N., Minamiguchi, S., Haga, H., et al. (2020) · medRxiv · doi:10.1101/2019.12.30.19016162 · Cited by 14 · see citing works
“In the research field on clinical decision support, collective intelligence approach has recently attracted attention, based on the reported higher diagnostic performance by several specialists than by a single specialist [42,43].”
BackgroundArmstrong, V., Tan, N., Sekhar, A. K., et al. (2019) · Academic Radiology · doi:10.1016/j.acra.2019.11.009 · Cited by 13 · see citing works
Khoong, E. C., Nouri, S. S., Tuot, D. S., Nundy, S., Fontil, V., & Sarkar, U. (2021) · Medical Decision Making · doi:10.1177/0272989x211031209 · Cited by 13 · see citing works
“results of CI collected from 1 platform are likely generalizable.(16,19,20) The users of Human Dx may not be representative of all physicians, but by removing medical students and international practitioners from our analysis, it is more likely these findings are generalizable to practicing primary care clinicians in…”
BackgroundMethodResultInfluentialSelf-citationExperimental evidence for structured information–sharing networks reducing medical errors
Centola, D., Becker, J., Zhang, J.-W., Aysola, J., Guilbeault, D., & Khoong, E. C. (2023) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.2108290120 · Cited by 20 · see citing works
Puissant, M. M., Giampalmo, S., Wira, C., et al. (2024) · Stroke · doi:10.1161/strokeaha.123.043406 · Cited by 13 · see citing works
Kachman, M. M., Carter, K., Arora, V. M., Flores, A., Meltzer, D. O., & Martin, S. K. (2020) · Journal of Hospital Medicine · doi:10.12788/jhm.3355 · Cited by 12 · see citing works
Why we succeed and fail in detecting fetal growth restriction: A population‐based study
Andreasen, L. A., Tabor, A., Nørgaard, L. N., Taksøe‐Vester, C., Krebs, L., Jørgensen, F. S., Jepsen, I. E., Sharif, H., Zingenberg, H., Rosthøj, S., et al. (2020) · Acta Obstetricia Et Gynecologica Scandinavica · doi:10.1111/aogs.14048 · Cited by 12 · see citing works
“Age, years (median, Q1 – Q3) 29 (27-33) 29 (26-34) 28 (25-32) 30 (27-34) 28 (24-32) 31 (28-35) 29 (25-34) BMI, kg/m2 (median, Q1 – Q3) 22 (20-25) 22 (20-25) 22 (20-25) 21 (20-24) 23 (20-26) 21 (20-24) 22 (20-25)”
Ngo, B., Nguyen, D., & vanSonnenberg, E. (2021) · MedEdPublish · doi:10.15694/mep.2021.000131.2 · Cited by 11 · see citing works
Hasan, E., Duhaime, E., & Trueblood, J. S. (2024) · Cognitive Research Principles and Implications · doi:10.1186/s41235-024-00558-6 · Cited by 9 · see citing works
“Even experts with extensive training are often wrong (Tschandl et al., 2019; Kämmer et al., 2017; Barnett et al., 2019).”
BackgroundDealing with expert bias in collective decision-making
Abels, A., Lenaerts, T., Trianni, V., & Nowé, A. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2023.103921 · Cited by 9 · see citing works
“support from a group of expert diagnosticians (Barnett et al., 2019).”
BackgroundGraziadio, S., Gregg, E. E., Allen, A. J., Neveux, P., Monz, B. U., Davenport, C. F., Mealing, S., Holmes, H., & di Ruffano, L. F. (2024) · Value in Health · doi:10.1016/j.jval.2024.02.003 · Cited by 9 · see citing works
Diagnostic Agreement Between Telemedicine on Social Networks and Teledermatology Centers
Serhrouchni, S., & Malmartel, A. (2021) · The Annals of Family Medicine · doi:10.1370/afm.2608 · Cited by 9 · see citing works
A diagnostic time-out to improve differential diagnosis in pediatric abdominal pain
Kasick, R. T., Melvin, J. E., Perera, S. T., Perry, M. F., Black, J. D., Bode, R. S., Groner, J. I., Kersey, K. E., Klamer, B. G., Bai, S., et al. (2019) · Diagnosis · doi:10.1515/dx-2019-0054 · Cited by 9 · see citing works
Schaller, M., Steinmetz, H., Vollmer, F. S., Plesac, M., Wicke, F. S., & Foerch, C. (2021) · Diagnosis · doi:10.1515/dx-2019-0108 · Cited by 9 · see citing works
Attaining Diagnostic Excellence
Bordini, B. J., Walsh, R. D., Basel, D., & Deshmukh, T. K. (2023) · Medical Clinics of North America · doi:10.1016/j.mcna.2023.06.013 · Cited by 8 · see citing works
Kaufmann, T., Forte, J. C., Hiemstra, B., Wiering, M., Grzegorczyk, M. A., Epema, A. H., van der Horst, I. C. C., & SICS Study Group (2019) · JMIR Medical Informatics · doi:10.2196/15358 · Cited by 8 · see citing works
Digitale Diagnoseunterstützung in der Rheumatologie
Knitza, J., Krusche, M., & Leipe, J. (2021) · Zeitschrift für Rheumatologie · doi:10.1007/s00393-021-01097-x · Cited by 8 · see citing works
How insight contributes to diagnostic excellence
Shimizu, T., & Graber, M. L. (2022) · Diagnosis · doi:10.1515/dx-2022-0007 · Cited by 8 · see citing works
Collective intelligence improves probabilistic diagnostic assessments
Stehouwer, N. R., Torrey, K. W., & Dell, M. S. (2023) · Diagnosis · doi:10.1515/dx-2022-0090 · Cited by 8 · see citing works
Lakhlifi, C., & Rohaut, B. (2023) · La Presse Médicale · doi:10.1016/j.lpm.2023.104181 · Cited by 8 · see citing works
Ronzio, L., Campagner, A., Cabitza, F., & Gensini, G. F. (2021) · Journal of Intelligence · doi:10.3390/jintelligence9020017 · Cited by 7 · see citing works
“…continuous feedback among the raters and their synchronous and co-located interaction and mutual influence Bahrami et al. (2010); Rosenberg et al. (2018), to approaches based on the aggregation of the opinions of different experts who do not directly interact with each other Barnett et al. (2019).”
BackgroundMethodShimizu, T. (2022) · Diagnostics · doi:10.3390/diagnostics12020356 · Cited by 7 · see citing works
“The usefulness of collective intelligence (CI) has long been noted [33].”
BackgroundArbic, N., Venet, M., Iriart, X., et al. (2022) · Frontiers in Pediatrics · doi:10.3389/fped.2022.891360 · Cited by 7 · see citing works
“Having access to expert echocardiographic skills can, however, be beneficial, especially for patients with complex heart disease (15).”
BackgroundAmzallag, J., Ropers, J., Shotar, E., Mathon, B., Jacquens, A., Degos, V., & Bernard, R. (2023) · Neurocritical Care · doi:10.1007/s12028-023-01718-0 · Cited by 7 · see citing works
“Knowing this, we used the mean of the two physician predictions as the most reliable estimate of physicians’ prediction, in accordance with the “wisdom of crowds” theory initiated by Francis Galton in 1907 [18] and subsequently found in other studies on patient prognosis [19, 20].”
MethodCollective Intelligence for Clinical Diagnosis—Are 2 (or 3) Heads Better Than 1?
Fihn, S. D. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.1071 · Cited by 7 · see citing works
The Regulatory Implications of Engaging Registered Nurses in Diagnoses
Cahill, M., Gleason, K. T., Harkless, G., Stanley, J. M., & Graber, M. L. (2019) · Journal of Nursing Regulation · doi:10.1016/s2155-8256(19)30110-3 · Cited by 7 · see citing works
Liao, M., & Tan, N. (2020) · Current Problems in Diagnostic Radiology · doi:10.1067/j.cpradiol.2020.09.017 · Cited by 7 · see citing works
Mosquera, C., Binder, F., Díaz, F. N., Seehaus, A., Ducrey, G., Ocantos, J., Aineseder, M., Rubin, L., Rabinovich, D., Quiroga, A. E., et al. (2021) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2021.100039 · Cited by 7 · see citing works
Using conversant artificial intelligence to improve diagnostic reasoning: ready for prime time?
Scott, I. A., Miller, T., & Crock, C. (2024) · The Medical Journal of Australia · doi:10.5694/mja2.52401 · Cited by 6 · see citing works
“29,30 The benefit of this collective intelligence can feasibly be replicated using LLMs.”
Whether two heads are better than one is the wrong question (though sometimes they are)
Hautz, W. E., Hautz, S. C., & Kämmer, J. E. (2020) · Advances in Health Sciences Education · doi:10.1007/s10459-020-09956-z · Cited by 6 · see citing works
Setting Assessment Standards for Artificial Intelligence Computer Vision Wound Annotations
Jopling, J. K., Pridgen, B. C., & Yeung, S. Y. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.7851 · Cited by 6 · see citing works
Unno, S., Igarashi, K., Saito, H., et al. (2022) · Endoscopy International Open · doi:10.1055/a-1922-6429 · Cited by 6 · see citing works
“This competency is related to the fact that collective intelligence can improve the diagnostic accuracy [17].”
BackgroundInfluentialHybrid Collective Intelligence for Decision Support in Complex Open-Ended Domains
Trianni, V., Nuzzolese, A. G., Porciello, J., et al. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230079 · Cited by 6 · see citing works
“, Ri(n) = n−1) [56,53], in this paper we search for optimal values by means of a brute-force approach, which we evaluated only for the case N = 10 and Nu = Nc = 5, allowing weights to vary in the interval [0,1].”
MethodSLEEPYLAND: trust begins with fair evaluation of automatic sleep staging models
Rossi, A. D., Metaldi, M., Bechný, M., Filchenko, I., van der Meer, J., Schmidt, M. H., Bassetti, C. L., Tzovara, A., Faraci, F. D., & Fiorillo, L. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02237-2 · Cited by 5 · see citing works
Medical residents’ perceptions of group biases in medical decision making: a qualitative study
Choi, J. J., Mhaimeed, N., Al-Mohanadi, D. H., & Mahmoud, M. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05643-4 · Cited by 6 · see citing works
Versorgernetzwerke für Menschen mit Seltenen Erkrankungen: Daten und Expertise bündeln
Graeßner, H., Storf, H., & Schaefer, F. S. (2022) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/s00103-022-03592-1 · Cited by 5 · see citing works
Song, X., Li, H., Chen, Q., et al. (2023) · Frontiers in Surgery · doi:10.3389/fsurg.2022.1060691 · Cited by 5 · see citing works
“Three or more clinicians will also combine their clinical experience to determine the likelihood of pneumonia in patients with flail chest (38).”
BackgroundAchangwa, N. R., Nierobisch, N., Ludovichetti, R., de Figueiredo, G. N., Kupka, M., Vere-Tyndall, A. D., Frauenfelder, T., Kulcsár, Z., & Hainc, N. (2024) · Current Problems in Diagnostic Radiology · doi:10.1067/j.cpradiol.2024.01.004 · Cited by 5 · see citing works
Rikhye, R. V., Hong, G., Singh, P., Smith, M. A., Loh, A., Muralidharan, V., Wong, D., Sayres, R., Phung, M., Betancourt, N. J., et al. (2024) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2024.01.005 · Cited by 5 · see citing works
Miyawaki, A., Burke, L. G., Khullar, D., & Tsugawa, Y. (2020) · Social Science & Medicine · doi:10.1016/j.socscimed.2020.113283 · Cited by 5 · see citing works
“Our outcomes of interest were (1) all-cause readmission, captured according to methodology recommended by the HCUP (Barrett et al., 2011), and (2) all-cause ED revisit within 30 days after hospital discharge, as prior studies have suggested that examining readmissions alone may not fully capture acute care utilization…”
MethodWinkler, J. K., Sies, K., Fink, C., Toberer, F., Enk, A., Abassi, M. S., Fuchs, T., Blum, A., Stolz, W., Coras‐Stepanek, B., et al. (2021) · JDDG Journal der Deutschen Dermatologischen Gesellschaft · doi:10.1111/ddg.14510_g · Cited by 5 · see citing works
Lau, Y.-T. K., Alemán, M. J., Medina, R., Brondfield, S., & Nematollahi, S. (2023) · Teaching and Learning in Medicine · doi:10.1080/10401334.2023.2226661 · Cited by 5 · see citing works
Training for team-based diagnosis: The future of interprofessional education
Danielson, J. H., Jones, M., Graber, M. L., Ballard, J., & Grice, G. R. (2022) · Journal of Interprofessional Education & Practice · doi:10.1016/j.xjep.2022.100551 · Cited by 5 · see citing works
A Narrative Review of Artificial Intelligence in Medical Diagnostics
Hirosawa, T., & Shimizu, T. (2025) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2025.063803 · Cited by 5 · see citing works
Diagnostic errors in patients admitted directly from new outpatient visits
Watanabe, Y., Miyagami, T., Shimizu, T., Nishizaki, Y., Ukishima, S., Santo, K., Kushiro, S., Aoki, N., Suzuki, M., Kanazawa, A., et al. (2024) · Diagnosis · doi:10.1515/dx-2024-0088 · Cited by 4 · see citing works
Zhang, X., Xie, Z., Xiang, Y., Baig, I. T., Kozman, M. M., Stender, C. F., Giancardo, L., & Tao, C. (2022) · JMIR Dermatology · doi:10.2196/39113 · Cited by 4 · see citing works
“Several studies have suggested that diagnoses derived using more than one source of input are more accurate than are those conceived by one method alone [27-29].”
BackgroundThe utility of indigenous knowledge in the pursuit of a UN sustainable development goal
Beckett, R. C., & O’Loughlin, A. (2024) · Collective Intelligence · doi:10.1177/26339137241241295 · Cited by 2 · see citing works
“The Wisdom of Crowds, Barnett et al., 2019).”
The Socrates Project for Difficult Diagnosis at Northwestern Medicine
Singer, B. D., Goodwin, A. M., Patel, A. A., & Vaughan, D. E. (2019) · Journal of Hospital Medicine · doi:10.12788/jhm.3335 · Cited by 4 · see citing works
Khoong, E. C., Fontil, V., Rivadeneira, N. A., et al. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa278 · Cited by 4 · see citing works
Self-citation
- Authors
- Barnett, M. L., Boddupalli, D., Nundy, S., & Bates, D. W.
- Affiliations
- Brigham and Women's Hospital (US)
- University of California, San Francisco (US)
- George Washington University (US)
- Corresponding author
- Barnett, Michael L.
- Venue
- JAMA Network Open
- Publisher
- American Medical Association (AMA)
- Published
- 2019-03-01
- Volume
- 2
- Issue
- 3
- Pages
- e190096
- ISSN
- 2574-3805 (ISSN-L)
- PMC
- PMC6484633
- OpenAlex
- W2919759422
- Semantic Scholar
- 7272c2e2748e6dea083da105c07c43a04795cbfa
- Cited by
- 154
- References
- 26
- Citations by year
- 2026: 12 · 2025: 13 · 2024: 23 · 2023: 20 · 2022: 19 · 2021: 22 · 2020: 31 · 2019: 14
- FWCI
- 22.12
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JAMA Network Open (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
- Keywords
- collective intelligence, diagnostic accuracy, misdiagnosis, group diagnosis, physician diagnosis, differential diagnosis, medical trainees, medical students, presenting symptoms, abdominal pain, diagnostic error reduction, clinical decision support, physician-patient communication, shared decision-making
- MeSH terms
- Cooperative Behavior, Cross-Sectional Studies, Diagnosis, Differential, Diagnostic Errors, Female, Humans, Intelligence, Male, Medical Staff, Hospital, Physicians, Professional Practice, Students, Medical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
