Must-have Qualities of Clinical Research on Artificial Intelligence and Machine Learning
Open accessAbstract
In the field of computer science, known as artificial intelligence, algorithms imitate reasoning tasks that are typically performed by humans. The techniques that allow machines to learn and get better at tasks such as recognition and prediction, which form the basis of clinical practice, are referred to as machine learning, which is a subfield of artificial intelligence. The number of artificial intelligence-and machine learnings-related publications in clinical journals has grown exponentially, driven by recent developments in computation and the accessibility of simple tools. However, clinicians are often not included in data science teams, which may limit the clinical relevance, explanability, workflow compatibility, and quality improvement of artificial intelligence solutions. Thus, this results in the language barrier between clinicians and artificial intelligence developers. Healthcare practitioners sometimes lack a basic understanding of artificial intelligence research because the approach is difficult for non-specialists to understand. Furthermore, many editors and reviewers of medical publications might not be familiar with the fundamental ideas behind these technologies, which may prevent journals from publishing high-quality artificial intelligence studies or, worse still, could allow for the publication of low-quality works. In this review, we aim to improve readers’ artificial intelligence literacy and critical thinking. As a result, we concentrated on what we consider the 10 most important qualities of artificial intelligence research: valid scientific purpose, high-quality data set, robust reference standard, robust input, no information leakage, optimal bias-variance tradeoff, proper model evaluation, proven clinical utility, transparent reporting, and open science. Before designing a study, one should have defined a sound scientific purpose. Then, it should be backed by a high-quality data set, robust input, and a solid reference standard. The artificial intelligence development pipeline should prevent information leakage. For the models, optimal bias-variance tradeoff should be achieved, and generalizability assessment must be adequately performed. The clinical value of the final models must also be established. After the study, thought should be given to transparency in publishing the process and results as well as open science for sharing data, code, and models. We hope this work may improve the artificial intelligence literacy and mindset of the readers.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Any flaw in the essential starting points of research question formulation inevitably generates bias in subsequent steps such as dataset selection, model development, and deployment.
“Any flaw in these essential starting points inevitably generates a bias in the subsequent steps, such as the selection of training datasets, AI model development, and/or deployment.”
From Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects (2024) · cites this paper as [22]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · see citing works
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450 · see citing works
Radiomics: the bridge between medical imaging and personalized medicine
Lambin, P., Leijenaar, R. T. H., Deist, T. M., Peerlings, J., de Jong, E. E. C., van Timmeren, J., Sanduleanu, S., Larue, R. T. H. M., Even, A. J. G., Jochems, A., et al. (2017) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2017.141 · Cited by 5,688 · see citing works
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616 · see citing works
The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository
Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., et al. (2013) · Journal of Digital Imaging · doi:10.1007/s10278-013-9622-7 · Cited by 4,599 · see citing works
Neural Networks and the Bias/Variance Dilemma
Geman, S., Bienenstock, E., & Doursat, R. (1992) · Neural Computation · doi:10.1162/neco.1992.4.1.1 · Cited by 3,578 · see citing works
International evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287 · see citing works
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212 · see citing works
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks
Papernot, N., McDaniel, P. D., Wu, X., Jha, S., & Swami, A. (2016) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2016.41 · Cited by 3,123 · see citing works
Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
Xu, W., Evans, D., & Qi, Y. (2018) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23198 · Cited by 1,859 · see citing works
Toward a Shared Vision for Cancer Genomic Data
Grossman, R. L., Heath, A. P., Ferretti, V., Varmus, H. E., Lowy, D. R., Kibbe, W. A., & Staudt, L. M. (2016) · New England Journal of Medicine · doi:10.1056/nejmp1607591 · Cited by 1,680 · see citing works
The Inevitable Application of Big Data to Health Care
Murdoch, T. B., & Detsky, A. S. (2013) · JAMA · doi:10.1001/jama.2013.393 · Cited by 1,676 · see citing works
Reconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · see citing works
Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443 · see citing works
Harmonization of multi-site diffusion tensor imaging data
Fortin, J.-P., Parker, D., Tunç, B., Watanabe, T., Elliott, M. A., Ruparel, K., Roalf, D. R., Satterthwaite, T. D., Gur, R. C., Gur, R. E., et al. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.08.047 · Cited by 1,261 · see citing works
The role of artificial intelligence in healthcare: a structured literature review
Secinaro, S. F., Calandra, D., Secinaro, A., Muthurangu, V., & Biancone, P. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01488-9 · Cited by 1,203 · see citing works
Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study
Chilamkurthy, S., Ghosh, R., Tanamala, S., Biviji, M., Campeau, N. G., Venugopal, V. K., Mahajan, V., Rao, P., & Warier, P. (2018) · The Lancet · doi:10.1016/s0140-6736(18)31645-3 · Cited by 1,020 · see citing works
Current Applications and Future Impact of Machine Learning in Radiology
Choy, G., Khalilzadeh, O., Michalski, M., Do, S., Samir, A. E., Pianykh, O. S., Geis, J. R., Pandharipande, P. V., Brink, J. A., & Dreyer, K. J. (2018) · Radiology · doi:10.1148/radiol.2018171820 · Cited by 800 · see citing works
Artificial intelligence in medical practice: the question to the answer?
Miller, D. D., & Brown, E. W. (2018) · The American Journal of Medicine · doi:10.1016/j.amjmed.2017.10.035 · Cited by 774 · see citing works
Model selection and overfitting
Lever, J., Krzywinski, M. I., & Altman, N. (2016) · Nature Methods · doi:10.1038/nmeth.3968 · Cited by 727 · see citing works
Artificial intelligence for clinical trial design
Harrer, S., Shah, P., Antony, B., & Hu, J. (2019) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2019.05.005 · Cited by 672 · see citing works
Measuring Computed Tomography Scanner Variability of Radiomics Features
Mackin, D., Fave, X. J., Zhang, L., Fried, D., Yang, J., Taylor, B. A., Rodriguez-Rivera, E., Dodge, C. T., Jones, A. K., & Court, L. E. (2015) · Investigative Radiology · doi:10.1097/rli.0000000000000180 · Cited by 649 · see citing works
Intrinsic dependencies of CT radiomic features on voxel size and number of gray levels
Shafiq‐ul‐Hassan, M., Zhang, G. G., Latifi, K., Ullah, G., Hunt, D., Balagurunathan, Y., Abdalah, M. A., Schabath, M. B., Goldgof, D. B., Mackin, D., et al. (2017) · Medical Physics · doi:10.1002/mp.12123 · Cited by 587 · see citing works
Vollmer, S. J., Mateen, B. A., Bohner, G., Király, F. J., Ghani, R., Jonsson, P. F., Cumbers, S., Jonas, A., McAllister, K., Myles, P. R., et al. (2020) · BMJ · doi:10.1136/bmj.l6927 · Cited by 554 · see citing works
Understanding adversarial attacks on deep learning based medical image analysis systems
Ma, X., Niu, Y., Gu, L., Wang, Y., Zhao, Y., Bailey, J., & Lu, F. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2020.107332 · Cited by 553 · see citing works
The “inconvenient truth” about AI in healthcare
Panch, T., Mattie, H., & Celi, L. A. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0155-4 · Cited by 473 · see citing works
van Leeuwen, K. G., Schalekamp, S., Rutten, M. J. C. M., van Ginneken, B., & de Rooij, M. (2021) · European Radiology · doi:10.1007/s00330-021-07892-z · Cited by 461 · see citing works
Reproducibility vs. Replicability: A Brief History of a Confused Terminology
Pleßer, H. E. (2018) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2017.00076 · Cited by 426 · see citing works
Handelman, G. S., Kok, H. K., Chandra, R. V., Razavi, A. H., Huang, S., Brooks, M., Lee, M. J., & Asadi, H. (2018) · American Journal of Roentgenology · doi:10.2214/ajr.18.20224 · Cited by 410 · see citing works
Bai, H. X., Wang, R., Xiong, Z., Hsieh, B., Chang, K., Halsey, K., Tran, T. M. L., Choi, J. W., Wang, D.-C., Shi, L.-B., et al. (2020) · Radiology · doi:10.1148/radiol.2020201491 · Cited by 381 · see citing works
Bluemke, D. A., Moy, L., Bredella, M. A., Ertl-Wagner, B. B., Fowler, K. J., Goh, V. J., Halpern, E. F., Hess, C. P., Schiebler, M. L., & Weiss, C. R. (2020) · Radiology · doi:10.1148/radiol.2019192515 · Cited by 343 · see citing works
How scientists fool themselves – and how they can stop
Nuzzo, R. (2015) · Nature · doi:10.1038/526182a · Cited by 296 · see citing works
The role of machine learning in clinical research: transforming the future of evidence generation
Weissler, E. H., Naumann, T., Andersson, T., Ranganath, R., Elemento, O., Luo, Y., Freitag, D. F., Benoit, J. R. A., Hughes, M. C., Khan, F. M., et al. (2021) · Trials · doi:10.1186/s13063-021-05489-x · Cited by 282 · see citing works
Assessing robustness of radiomic features by image perturbation
Zwanenburg, A., Leger, S., Agolli, L., Pilz, K., Troost, E. G. C., Richter, C., & Löck, S. (2019) · Scientific Reports · doi:10.1038/s41598-018-36938-4 · Cited by 258 · see citing works
Meyer, M., Ronald, J., Vernuccio, F., Nelson, R. C., Ramirez-Giraldo, J. C., Solomon, J., Patel, B. N., Samei, E., & Marin, D. (2019) · Radiology · doi:10.1148/radiol.2019190928 · Cited by 258 · see citing works
Exploring Large-scale Public Medical Image Datasets
Oakden-Rayner, L. (2020) · Academic Radiology · doi:10.1016/j.acra.2019.10.006 · Cited by 250 · see citing works
Sim, Y., Chung, M. J., Kotter, E., Yune, S., Kim, M., Do, S., Han, K., Kim, H., Yang, S., Lee, D.-J., et al. (2020) · Radiology · doi:10.1148/radiol.2019182465 · Cited by 248 · see citing works
Randomized Clinical Trials of Machine Learning Interventions in Health Care: A Systematic Review
Plana, D., Shung, D. L., Grimshaw, A. A., Saraf, A., Sung, J. J. Y., & Kann, B. H. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.33946 · Cited by 236 · see citing works
Could AI help you to write your next paper?
Hutson, M. (2022) · Nature · doi:10.1038/d41586-022-03479-w · Cited by 236 · see citing works
FIVES: A Fundus Image Dataset for Artificial Intelligence based Vessel Segmentation
Jin, K., Huang, X., Zhou, J., Li, Y., Yan, Y., Sun, Y., Zhang, Q., Wang, Y., & Ye, J. (2022) · Scientific Data · doi:10.1038/s41597-022-01564-3 · Cited by 233 · see citing works
Sengupta, P. P., Shrestha, S., Berthon, B., Messas, E., Donal, E., Tison, G. H., Min, J. K., D’hooge, J., Voigt, J., Dudley, J. T., et al. (2020) · JACC. Cardiovascular imaging · doi:10.1016/j.jcmg.2020.07.015 · Cited by 229 · see citing works
Mali, S. A., Ibrahim, A., Woodruff, H. C., Andrearczyk, V., Müller, H., Primakov, S. P., Salahuddin, Z., Chatterjee, A., & Lambin, P. (2021) · Journal of Personalized Medicine · doi:10.3390/jpm11090842 · Cited by 218 · see citing works
The security of machine learning in an adversarial setting: A survey
Wang, X., Li, J., Kuang, X., Tan, Y., & Li, J. (2019) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2019.03.003 · Cited by 197 · see citing works
Test–Retest Data for Radiomics Feature Stability Analysis: Generalizable or Study-Specific?
van Timmeren, J. E., Leijenaar, R. T. H., van Elmpt, W., Wang, J., Zhang, Z., Dekker, A. L. A. J., & Lambin, P. (2016) · Tomography · doi:10.18383/j.tom.2016.00208 · Cited by 191 · see citing works
Gray-level discretization impacts reproducible MRI radiomics texture features
Duron, L., Balvay, D., Perre, S. V., Bouchouicha, A., Savatovsky, J., Sadik, J., Thomassin‐Naggara, I., Fournier, L. S., & Lecler, A. (2019) · PLoS ONE · doi:10.1371/journal.pone.0213459 · Cited by 189 · see citing works
The DECIDE-AI Steering Group (2021) · Nature Medicine · doi:10.1038/s41591-021-01229-5 · Cited by 184 · see citing works
Deep learning-based unlearning of dataset bias for mri harmonisation and confound removal
Dinsdale, N. K., Jenkinson, M., & Namburete, A. I. L. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2020.117689 · Cited by 178 · see citing works
Data Analysis Strategies in Medical Imaging
Parmar, C., Barry, J. D., Hosny, A., Quackenbush, J. F., & Aerts, H. J. W. L. (2018) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-18-0385 · Cited by 161 · see citing works
Universal adversarial attacks on deep neural networks for medical image classification
Hirano, H., Minagi, A., & Takemoto, K. (2021) · BMC Medical Imaging · doi:10.1186/s12880-020-00530-y · Cited by 160 · see citing works
Clinician checklist for assessing suitability of machine learning applications in healthcare
Scott, I. A., Carter, S. M., & Coiera, E. W. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2020-100251 · Cited by 140 · see citing works
Structural Analysis and Optimization of Convolutional Neural Networks with a Small Sample Size
D’souza, R. N., Huang, P.-Y., & Yeh, F.-C. F. (2020) · Scientific Reports · doi:10.1038/s41598-020-57866-2 · Cited by 135 · see citing works
van de Sande, D., Van Genderen, M. E., Smit, J. M., Huiskens, J., Visser, J. J., Veen, R. E. R., van Unen, E., BA, O. H., Gommers, D., & Bommel, J. V. (2022) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100495 · Cited by 135 · see citing works
A decade of radiomics research: are images really data or just patterns in the noise?
dos Santos, D. P., Dietzel, M., & Baeßler, B. (2020) · European Radiology · doi:10.1007/s00330-020-07108-w · Cited by 129 · see citing works
Self-citationEnsuring that biomedical AI benefits diverse populations
Zou, J., & Schiebinger, L. (2021) · EBioMedicine · doi:10.1016/j.ebiom.2021.103358 · Cited by 124 · see citing works
Spadarella, G., Stanzione, A., D’Antonoli, T. A., Андрейченко, А. Е., Fanni, S. C., Ugga, L., Kotter, E., & Cuocolo, R. (2022) · European Radiology · doi:10.1007/s00330-022-09187-3 · Cited by 120 · see citing works
Self-citationBridging the chasm between AI and clinical implementation
Aristidou, A., Jena, R., & Topol, E. J. (2022) · The Lancet · doi:10.1016/s0140-6736(22)00235-5 · Cited by 115 · see citing works
A primer for understanding radiology articles about machine learning and deep learning
Nakaura, T., Higaki, T., Awai, K., Ikeda, O., & Yamashita, Y. (2020) · Diagnostic and Interventional Imaging · doi:10.1016/j.diii.2020.10.001 · Cited by 107 · see citing works
An, C., Park, Y. W., Ahn, S. S., Han, K., Kim, H., & Lee, S. (2021) · PLoS ONE · doi:10.1371/journal.pone.0256152 · Cited by 107 · see citing works
How AI technology can tame the scientific literature
Extance, A. (2018) · Nature · doi:10.1038/d41586-018-06617-5 · Cited by 105 · see citing works
Standardization of imaging features for radiomics analysis
Haga, A., Takahashi, W., Aoki, S., Nawa, K., Yamashita, H., Abe, O., & Nakagawa, K. (2019) · The Journal of Medical Investigation · doi:10.2152/jmi.66.35 · Cited by 92 · see citing works
Formulating a good research question: Pearls and pitfalls
Fandino, W. (2019) · Indian Journal of Anaesthesia · doi:10.4103/ija.ija_198_19 · Cited by 89 · see citing works
Latent space manipulation for high-resolution medical image synthesis via the StyleGAN
Fetty, L., Bylund, M., Kuess, P., Heilemann, G., Nyholm, T., Georg, D., & Löfstedt, T. (2020) · Zeitschrift für Medizinische Physik · doi:10.1016/j.zemedi.2020.05.001 · Cited by 88 · see citing works
Cuocolo, R., Stanzione, A., Castaldo, A., Lucia, D. R. D., & Imbriaco, M. (2021) · European Journal of Radiology · doi:10.1016/j.ejrad.2021.109647 · Cited by 86 · see citing works
Self-citationAssessing the Economic Value of Clinical Artificial Intelligence: Challenges and Opportunities
Hendrix, N., Veenstra, D. L., Cheng, M. M., Anderson, N. C., & Verguet, S. (2021) · Value in Health · doi:10.1016/j.jval.2021.08.015 · Cited by 85 · see citing works
Mitigating Bias in Radiology Machine Learning: 2. Model Development
Zhang, K., Khosravi, B., Vahdati, S., Faghani, S., Nugen, F., Rassoulinejad-Mousavi, S. M., Moassefi, M., Jagtap, J. M. M., Singh, Y., Rouzrokh, P., et al. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.220010 · Cited by 83 · see citing works
Koçak, B., Kuş, E. A., Durmaz, E. Ş., Ulusan, M. B., & Kılıçkesmez, Ö. (2019) · European Radiology · doi:10.1007/s00330-019-6003-8 · Cited by 81 · see citing works
Self-citationLow replicability can support robust and efficient science
Lewandowsky, S., & Oberauer, K. (2020) · Nature Communications · doi:10.1038/s41467-019-14203-0 · Cited by 80 · see citing works
Jamthikar, A. D., Gupta, D., Mantella, L. E., Saba, L., Laird, J. R., Johri, A. M., & Suri, J. S. (2020) · International journal of cardiac imaging · doi:10.1007/s10554-020-02099-7 · Cited by 80 · see citing works
Fedorov, A., Longabaugh, W. J. R., Pot, D., Clunie, D., Pieper, S., Aerts, H. J. W. L., Homeyer, A., Lewis, R., Akbarzadeh, A., Bontempi, D., et al. (2021) · Cancer Research · doi:10.1158/0008-5472.can-21-0950 · Cited by 77 · see citing works
Fang, Y., Wang, J., Ou, X., Ying, H., Hu, C., Zhang, Z., & Hu, W. (2021) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ac2206 · Cited by 76 · see citing works
Sunoqrot, M. R. S., Saha, A., Hosseinzadeh, M., Elschot, M., & Huisman, H. (2022) · European Radiology Experimental · doi:10.1186/s41747-022-00288-8 · Cited by 76 · see citing works
Improved CT-based Osteoporosis Assessment with a Fully Automated Deep Learning Tool
Pickhardt, P. J., Nguyễn, T. V., Perez, A. A., Graffy, P. M., Jang, S., Summers, R. M., & Garrett, J. W. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.220042 · Cited by 76 · see citing works
Penzkofer, T., Padhani, A. R., Türkbey, B., Haider, M. A., Huisman, H. J., Walz, J. C., Salomon, G., Schoots, I. G., Richenberg, J., Villeirs, G. M., et al. (2021) · European Radiology · doi:10.1007/s00330-021-08021-6 · Cited by 73 · see citing works
Romeo, V., Maurea, S., Cuocolo, R., Petretta, M., Mainenti, P. P., Verde, F., Coppola, M., Dell'Aversana, S., & Brunetti, A. (2018) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.25954 · Cited by 69 · see citing works
Self-citationMRI image harmonization using cycle-consistent generative adversarial network
Modanwal, G., Vellal, A. D., Buda, M., & Mazurowski, M. A. (2020) · SPIE Medical Imaging: Computer-Aided Diagnosis · doi:10.1117/12.2551301 · Cited by 67 · see citing works
Koçak, B., Durmaz, E. Ş., Kaya, O. K., Kuş, E. A., & Kılıçkesmez, Ö. (2019) · American Journal of Roentgenology · doi:10.2214/ajr.19.21212 · Cited by 64 · see citing works
Self-citationOncologic Imaging and Radiomics: A Walkthrough Review of Methodological Challenges
Stanzione, A., Cuocolo, R., Ugga, L., Verde, F., Romeo, V., Brunetti, A., & Maurea, S. (2022) · Cancers · doi:10.3390/cancers14194871 · Cited by 64 · see citing works
Self-citationSanford, T., Zhang, L., Harmon, S. A., Sackett, J., Yang, D., Roth, H. R., Xu, Z., Kesani, D., Mehralivand, S., Baroni, R. H., et al. (2020) · American Journal of Roentgenology · doi:10.2214/ajr.19.22347 · Cited by 62 · see citing works
Al‐Zaiti, S. S., Alghwiri, A. A., Hu, X., Clermont, G., Peace, A. J., Macfarlane, P. W., & Bond, R. (2022) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztac016 · Cited by 61 · see citing works
Pandiyan, S., & Wang, L. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.106140 · Cited by 60 · see citing works
Picking a Research Problem -- The Critical Decision
Kahn, C. R. (1994) · New England Journal of Medicine · doi:10.1056/nejm199405263302113 · Cited by 59 · see citing works
Open science, data sharing and solidarity: who benefits?
Staunton, C., Barragán, C. A., Canali, S., Ho, C. W.-L., Leonelli, S., Mayernik, M. S., Prainsack, B., & Wonkham, A. (2021) · History & Philosophy of the Life Sciences · doi:10.1007/s40656-021-00468-6 · Cited by 57 · see citing works
Biosecurity in an age of open science
Smith, J. A., & Sandbrink, J. B. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001600 · Cited by 57 · see citing works
Johri, A. M., Singh, K. V., Mantella, L. E., Saba, L., Sharma, A., Laird, J. R., Utkarsh, K., Singh, I. M., Gupta, S., Kalra, M. S., et al. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.106018 · Cited by 50 · see citing works
Mascalzoni, D., Bentzen, H. B., Budin‐Ljøsne, I., Bygrave, L. A., Bell, J., Dove, E. S., Fuchsberger, C., Hveem, K., Mayrhofer, M. T., Meraviglia, V., et al. (2019) · Annals of Internal Medicine · doi:10.7326/m18-2854 · Cited by 49 · see citing works
Lee, H., Huang, C., Yune, S., Tajmir, S., Kim, M., & Do, S. (2019) · Scientific Reports · doi:10.1038/s41598-019-51779-5 · Cited by 45 · see citing works
Scoping review of the current landscape of AI-based applications in clinical trials
Cascini, F., Beccia, F., Causio, F. A., Melnyk, A., Zaino, A., & Ricciardi, W. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.949377 · Cited by 45 · see citing works
Lee, S., Elton, D. C., Yang, A. H., Koh, C., Kleiner, D. E., Lubner, M. G., Pickhardt, P. J., & Summers, R. M. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.210268 · Cited by 44 · see citing works
On the role of deep learning model complexity in adversarial robustness for medical images
Rodriguez, D., Nayak, T., Chen, Y., Krishnan, R., & Huang, Y. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-01891-w · Cited by 43 · see citing works
From ideas to studies: how to get ideas and sharpen them into research questions
Vandenbroucke, J. P., & Pearce, N. E. (2018) · Clinical Epidemiology · doi:10.2147/clep.s142940 · Cited by 39 · see citing works
Hullman, J. R., Kapoor, S., Nanayakkara, P., Gelman, A., & Narayanan, A. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534196 · Cited by 37 · see citing works
Wang, H., Liu, Y., Xu, N., Sun, Y., Fu, S., Wu, Y., Liu, C., Cui, L., Liu, Z., Chang, Z., et al. (2022) · European Journal of Radiology · doi:10.1016/j.ejrad.2022.110527 · Cited by 36 · see citing works
Allen, R., & Masters, D. (2020) · ERA Forum · doi:10.1007/s12027-019-00582-w · Cited by 34 · see citing works
Helman, S., Terry, M. A., Pellathy, T., Williams, A., Dubrawski, A., Clermont, G., Pinsky, M. R., Al-Zaiti, S., & Hravnak, M. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104643 · Cited by 31 · see citing works
Machine learning models predict coagulopathy in spontaneous intracerebral hemorrhage patients in ER
Zhu, F., Pan, Z., Tang, Y., Fu, P., Cheng, S., Hou, W., Zhang, Q., Huang, H., & Sun, Y. (2020) · CNS Neuroscience & Therapeutics · doi:10.1111/cns.13509 · Cited by 30 · see citing works
Sachs, P. B., Hunt, K., Mansoubi, F., & Borgstede, J. P. (2016) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-016-9895-8 · Cited by 29 · see citing works
Masson, I., Da‐ano, R., Lucia, F., Dore, M., Castelli, J., de Monsabert, C. G., Ramée, J., Sellami, S., Visvikis, D., Hatt, M., et al. (2021) · Medical Physics · doi:10.1002/mp.14948 · Cited by 26 · see citing works
Data Quality, Data Sharing, and Moving Artificial Intelligence Forward
Elmore, J. G., & Lee, C. I. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.19345 · Cited by 22 · see citing works
Li, T., Huang, H., Zhang, S., Zhang, Y., Jing, H., Sun, T., Zhang, X., Lu, L., & Zhang, M. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.984750 · Cited by 20 · see citing works
A Deep Learning Pipeline for Nucleus Segmentation
Zaki, G., Gudla, P. R., Lee, K., Kim, J. J., Ozbun, L. L., Shachar, S., Gadkari, M., Sun, J., Fraser, I. D. C., Franco, L. M., et al. (2020) · Cytometry Part A · doi:10.1002/cyto.a.24257 · Cited by 19 · see citing works
Current status of clinical research using artificial intelligence techniques
Karekar, S. R., & Vazifdar, A. K. (2021) · Perspectives in Clinical Research · doi:10.4103/picr.picr_25_20 · Cited by 17 · see citing works
Deng, J., & He, Z. (2022) · Frontiers in Surgery · doi:10.3389/fsurg.2022.891984 · Cited by 10 · see citing works
Interpretation of artificial intelligence studies for the ophthalmologist
Tan, T., Xu, X., Wang, Z., Liu, Y., & Ting, D. S. W. (2020) · Current Opinion in Ophthalmology · doi:10.1097/icu.0000000000000695 · Cited by 8 · see citing works
(2020) · Nature Cancer · doi:10.1038/s43018-020-00151-y · Cited by 8 · see citing works
Hu, G., Hu, X., Yang, K., Yu, Y., Jiang, Z., Liu, Y., Liu, D., Hu, X., Xiao, H., Zou, Y., et al. (2022) · Journal of Computer Assisted Tomography · doi:10.1097/rct.0000000000001386 · Cited by 8 · see citing works
Mottaghy, F. M., Hertel, F., & Beheshti, M. (2021) · Methods · doi:10.1016/j.ymeth.2021.02.011 · Cited by 6 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [1]Goodfellow I, Bengio Y, Courville A. Deep Learning. MIT Press; 2016. at [Internet] http://www.deeplearningbook.org
- [18]Pineau J Vincent-Lamarre P Sinha K et al Improving Reproducibility in Machine Learning Research(A Report from the NeurIPS 2019 Reproducibility Program) J Mach Learn Res. 2021 22 1 20
Cited by
Sorted by citations
Javed, H., El–Sappagh, S., & Abuhmed, T. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-11005-9 · Cited by 205 · see citing works
Koçak, B., Ponsiglione, A., Stanzione, A., Bluethgen, C., Santinha, J., Ugga, L., Huisman, M., Klontzas, M. E., Cannella, R., & Cuocolo, R. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.242854 · Cited by 168 · see citing works
Self-citationThe Role of Artificial Intelligence in Coronary Artery Disease and Atrial Fibrillation
Hayıroğlu, M. İ., & Altay, S. (2023) · Balkan Medical Journal · doi:10.4274/balkanmedj.galenos.2023.06042023 · Cited by 116 · see citing works
Owusu-Adjei, M., Hayfron-Acquah, J. B., Twum, F., & Abdul-Salaam, G. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000290 · Cited by 116 · see citing works
Xiao, B., Li, S., Dou, S., He, H., Fu, B., Zhang, T., Sun, W., Yang, Y., Xiong, Y., Shi, J., et al. (2023) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2023.108559 · Cited by 61 · see citing works
Imaging Diagnosis of Hepatocellular Carcinoma: A State-of-the-Art Review
Candita, G., Rossi, S., Cwiklinska, K., Fanni, S. C., Cioni, D., Lencioni, R. A., & Neri, E. (2023) · Diagnostics · doi:10.3390/diagnostics13040625 · Cited by 59 · see citing works
Integration of artificial intelligence in clinical laboratory medicine: Advancements and challenges
Xie, H., Jia, Y. H., & Liu, S. (2024) · Interdisciplinary medicine · doi:10.1002/inmd.20230056 · Cited by 52 · see citing works
Radiology AI and sustainability paradox: environmental, economic, and social dimensions
Koçak, B., Ponsiglione, A., Romeo, V., Ugga, L., Huisman, M., & Cuocolo, R. (2025) · Insights into Imaging · doi:10.1186/s13244-025-01962-2 · Cited by 58 · see citing works
Self-citationDelta-radiomics in cancer immunotherapy response prediction: A systematic review
Abbas, E., Fanni, S. C., Bandini, C., Francischello, R., Febi, M., Aghakhanyan, G., Ambrosini, I., Faggioni, L., Cioni, D., Lencioni, R. A., et al. (2023) · European Journal of Radiology Open · doi:10.1016/j.ejro.2023.100511 · Cited by 38 · see citing works
Yun, S. (2024) · Journal of Yeungnam Medical Science · doi:10.12701/jyms.2024.00668 · Cited by 34 · see citing works
Ponsiglione, A., Gambardella, M., Stanzione, A., Green, R., Cantoni, V., Nappi, C., Crocetto, F., Cuocolo, R., Cuocolo, A., & Imbriaco, M. (2023) · European Radiology · doi:10.1007/s00330-023-10427-3 · Cited by 34 · see citing works
Self-citationAdigwe, O. P., Onavbavba, G., & Sanyaolu, S. E. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1293297 · Cited by 32 · see citing works
Romagnoli, A., Ferrara, F., Langella, R., & Zovi, A. (2024) · Pharmaceutical Research · doi:10.1007/s11095-024-03685-3 · Cited by 28 · see citing works
Maniscalco, M., Candia, C., Fuschillo, S., Ambrosino, P., Paris, D., & Motta, A. (2024) · Journal of Breath Research · doi:10.1088/1752-7163/ad7a9a · Cited by 22 · see citing works
Predicting Coronary Artery Disease Utilizing Support Vector Machines: Optimizing Predictive Model
Khazaal, S. M., & Maarouf, H. (2023) · Mesopotamian Journal of Artificial Intelligence in Healthcare · doi:10.58496/mjaih/2023/004 · Cited by 19 · see citing works
Implications of Data Leakage in Machine Learning Preprocessing: A Multi-Domain InvestigationPreprint
Bouke, M. A., Zaid, S. A., & Abdullah, A. (2024) · Research Square · doi:10.21203/rs.3.rs-4579465/v1 · Cited by 20 · see citing works
Hybrid Machine Learning Approach to Zero-Inflated Data Improves Accuracy of Dengue Prediction
Francisco, M. E., Carvajal, T. M., & Watanabe, K. (2024) · PLoS neglected tropical diseases · doi:10.1371/journal.pntd.0012599 · Cited by 17 · see citing works
Artificial Intelligence and Radiomics for Endometrial Cancer MRI: Exploring the Whats, Whys and Hows
Leo, E., Stanzione, A., Miele, M., Cuocolo, R., Sica, G., Scaglione, M., Camera, L., Maurea, S., & Mainenti, P. P. (2023) · Journal of Clinical Medicine · doi:10.3390/jcm13010226 · Cited by 16 · see citing works
Self-citationKoçak, B., dos Santos, D. P., & Dietzel, M. (2025) · European Journal of Radiology Artificial Intelligence · doi:10.1016/j.ejrai.2025.100004 · Cited by 16 · see citing works
Self-citationRegenerative bioelectronics: A strategic roadmap for precision medicine
Panda, A. K., & Basu, B. (2023) · Biomaterials · doi:10.1016/j.biomaterials.2023.122271 · Cited by 15 · see citing works
Erdoğan, S. (2024) · Balkan Medical Journal · doi:10.4274/balkanmedj.galenos.2024.2024-080824 · Cited by 13 · see citing works
Banchero, L., Vacalebri, F., Mossi, J. M., & López, J. J. (2025) · Sensors · doi:10.3390/s25030793 · Cited by 13 · see citing works
Management of polypharmacy through deprescribing in older patients: a review of the role of AI tools
Meslamani, A. Z. A. (2025) · Expert Review of Clinical Pharmacology · doi:10.1080/17512433.2025.2519648 · Cited by 10 · see citing works
AI in plastic surgery: customizing care for each patient
Brenac, C., Fazilat, A. Z., Fallah, M., Kawamoto-Duran, D., Sunwoo, P. S., Longaker, M. T., Wan, D. C., & Guo, J. L. (2024) · Artificial Intelligence Surgery · doi:10.20517/ais.2024.49 · Cited by 13 · see citing works
Zhang, X., Wang, X., Shi, S., & Guo, D. (2025) · Biomarker Research · doi:10.1186/s40364-025-00854-3 · Cited by 11 · see citing works
A first look into radiomics application in testicular imaging: A systematic review
Fanni, S. C., Febi, M., Colligiani, L., Volpi, F., Ambrosini, I., Tumminello, L., Aghakhanyan, G., Aringhieri, G., Cioni, D., & Neri, E. (2023) · Frontiers in Radiology · doi:10.3389/fradi.2023.1141499 · Cited by 10 · see citing works
Robustness and Reliability Testing in Healthcare Using Artificial Intelligence
Khinvasara, T., Shankar, A., & Wong, C. (2024) · Asian Journal of Research in Computer Science · doi:10.9734/ajrcos/2024/v17i7482 · Cited by 11 · see citing works
Mehri-Kakavand, G., Mdletshe, S., & Wang, A. (2025) · Journal of Medical Radiation Sciences · doi:10.1002/jmrs.860 · Cited by 10 · see citing works
Sav, N. M. (2025) · Pediatric Nephrology · doi:10.1007/s00467-025-06723-3 · Cited by 9 · see citing works
Tiong, K. Y., Ma, Z., & Palmqvist, C.-W. (2025) · European Transport Research Review · doi:10.1186/s12544-024-00704-7 · Cited by 6 · see citing works
Use of artificial intelligence in animal experimentation: A review
Gonçalves, S. D., Rodrigues, M., Avidos, M., Morais, A. B., & Caramelo, A. (2025) · Toxicology Letters · doi:10.1016/j.toxlet.2025.07.1417 · Cited by 7 · see citing works
Radiomics Applications in Spleen Imaging: A Systematic Review and Methodological Quality Assessment
Fanni, S. C., Febi, M., Francischello, R., Caputo, F. P., Ambrosini, I., Sica, G., Faggioni, L., Masala, S. A., Tonerini, M., Scaglione, M., et al. (2023) · Diagnostics · doi:10.3390/diagnostics13162623 · Cited by 6 · see citing works
Abbas, S., Iftikhar, M., Shah, M. M., & Khan, S. J. (2024) · Cureus · doi:10.7759/cureus.75851 · Cited by 5 · see citing works
Diagnosing facial synkinesis using artificial intelligence to advance facial palsy care
Knoedler, L., Festbaum, C., Dean, J., Baecher, H., de Lambertye, G., Maul, M., Schaschinger, T., Niederegger, T., Scheiflinger, A., Alfertshofer, M. G., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-08548-4 · Cited by 5 · see citing works
Owusu-Adjei, M., Hayfron-Acquah, J. B., Twum, F., & Abdul-Salaam, G. (2023) · medRxiv · doi:10.1101/2023.06.01.23290837 · Cited by 5 · see citing works
de Mello, M. E. F., Rocha, É. D. S., & Endo, P. T. (2025) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-025-08028-7 · Cited by 4 · see citing works
Zhang, Y., Kirchler, T., & Wang, A. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02376-3 · Cited by 6 · see citing works
Estimating very low ejection fraction from the 12 Lead ECG among patients with acute heart failure
Bhattarai, S. P., Dzikowicz, D. J., Xue, Y., Block, R., Tucker, R., Bhandari, S., Boulware, V. E., Stone, B., & Carey, M. G. (2025) · Journal of Electrocardiology · doi:10.1016/j.jelectrocard.2025.153878 · Cited by 4 · see citing works
Present and Future of Artificial Intelligence in Pathology
Usta, U., & Taştekin, E. (2024) · Balkan Medical Journal · doi:10.4274/balkanmedj.galenos.2024.2024.060324 · Cited by 4 · see citing works
Current Applications of AI in Medical Imaging
Salle, G. D., Fanni, S. C., Aghakhanyan, G., & Neri, E. (2023) · Imaging informatics for healthcare professionals · doi:10.1007/978-3-031-25928-9_8 · Cited by 3 · see citing works
Al-Hammad, W. E., Kuroda, M., ALJAMAL, G., Fujikura, M., Kamizaki, R., Kuroda, K., Yoshida, S., Nakamura, Y., Oita, M., Tanabe, Y., et al. (2025) · Diagnostics · doi:10.3390/diagnostics15060668 · Cited by 2 · see citing works
Top 10 Tips for Writing about AI in Radiology: A Brief Guide for Authors
Atzen, S. L. (2025) · Radiology · doi:10.1148/radiol.243347 · Cited by 2 · see citing works
Statistical Rigor and Reproducibility in the AI Era
Korkmaz, S. (2025) · Balkan Medical Journal · doi:10.4274/balkanmedj.galenos.2025.2025.040825 · Cited by 2 · see citing works
Predicting Microwave Ablation Early Efficacy in Pulmonary Malignancies via Δ Radiomics Models
Yang, J., Yang, C., Feng, J., Zhu, F., & Zhao, Z. (2024) · Journal of Computer Assisted Tomography · doi:10.1097/rct.0000000000001611 · Cited by 1 · see citing works
Artificial Intelligence in Cardiovascular Healthcare
Dutta, U., & Das, B. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-9735-0.ch002 · Cited by 1 · see citing works
The True Value of Quantitative Imaging for Adrenal Mass Characterization: Reality or Possibility?
Stanzione, A., Romeo, V., & Maurea, S. (2023) · Cancers · doi:10.3390/cancers15020522 · Cited by 1 · see citing works
Self-citationEstimating Ejection Fraction from the 12 Lead ECG among Patients with Acute Heart FailurePreprint
Bhattarai, S. P., Dzikowicz, D. J., Xue, Y., Block, R., Tucker, R., Bhandari, S., Boulware, V. E., Stone, B., & Carey, M. G. (2024) · medRxiv · doi:10.1101/2024.03.25.24304875 · Cited by 1 · see citing works
Artificial Intelligence in Medicine for Advance Treatment: A Survey
Shnawa, A. H., Hadi, M. R., Jebur, M. A., Adnan, M. M., Hameed, W., & Mohammed, G. (2023) · International Conference on Engineering Technology and its Applications (IICETA) · doi:10.1109/iiceta57613.2023.10351440
Delta Radiomics — Potential Role in Head Neck CancerPreprint
Mali, S. B. (2024) · Qeios · doi:10.32388/35jd6d
Delta Radiomics — Potential Role in Head Neck CancerPreprint
Mali, S. B. (2024) · Qeios · doi:10.32388/35jd6d.2
Delta Radiomics — Potential role in Head Neck Cancer
Mali, S. B. (2024) · Oral Oncology Reports · doi:10.1016/j.oor.2024.100676
AI Integration in Medical Imaging: Advanced Analysis of Chest X-ray
Bandara, D., Sutharssan, T., & Sharif, M. S. (2024) · International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies (3ICT) · doi:10.1109/3ict64318.2024.10824281
Taylan, G., & ALTAY, S. (2025) · Namık Kemal Tıp Dergisi · doi:10.4274/nkmj.galenos.2024.57442
Dunmade, A. O., & Utulu, S. C. A. (2025) · De Gruyter eBooks · doi:10.1515/9783111176345-007
Misiurek, K., Misiurek, B., & Szyłko-Bigus, O. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5406514
Cuocolo, R., Stanzione, A., Ponsiglione, A., Ugga, L., & Koçak, B. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-29242-2.00012-4
Self-citationWu, B. (2025) · International Conference on Education Technology and Computer · doi:10.1109/icetc66579.2025.11387331
Singh, S., Ayeni, F. E., Sritharan, N., Riffat, F., Suruliraj, A., & Paramaesvaran, S. (2026) · Australian Journal of Otolaryngology · doi:10.21037/ajo-25-48
A Multi-Branch BiLSTM with Multi-Head Self-Attention for Suspicious Sound Recognition
Yusuf, S. M., Saidu, H., & Saminu, S. S. (2026) · Journal of Computing Theories and Applications · doi:10.62411/jcta.15777 · Cited by 1 · see citing works
Khoo, L. Y., & Dhillon, S. K. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag249
Diagnosis of Portal Hypertension: Advancing Towards Non‐Invasive Solutions
Yin, L., Chu, H., & Yang, L. (2026) · Portal hypertension & cirrhosis · doi:10.1002/poh2.70060
Zhang, W., Luo, Z., Liu, K., Wang, Y., Hou, Y., Su, X., Wu, X., Miao, C., Liu, Q., He, Z., et al. (2026) · 中国科学通报：英文版 · doi:10.1016/j.scib.2026.09.016
Cellina, M. I., Cacioppa, L. M., Bagnacci, G., Zamboni, G., Canto, M. T. D., Foti, G., Gioia, A. L., Bruno, A., Ippolito, D., Silipigni, S., et al. (2026) · La radiologia medica · doi:10.1007/s11547-026-02294-x
- Authors
- Koçak, B., Cuocolo, R., Santos, D. P. D., Stanzione, A., & Ugga, L.
- Affiliations
- Sağlık Bilimleri Üniversitesi (TR)
- University of Salerno (IT)
- University Hospital Frankfurt (DE)
- University of Naples Federico II (IT)
- Corresponding author
- Koçak, Burak
- Venue
- Balkan Medical Journal
- Publisher
- Galenos Yayinevi
- Published
- 2023-01-24
- Volume
- 40
- Issue
- 1
- Pages
- 3-12
- ISSN
- 1301-3149 (ISSN-L)
- 2146-3123
- 2146-3131
- PMC
- PMC9874249
- OpenAlex
- W4313280127
- Cited by
- 64
- References
- 106
- Citations by year
- 2026: 7 · 2025: 22 · 2024: 20 · 2023: 15
- FWCI
- 0.98
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by-nc-nd
- Locations
- Balkan Medical Journal (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Archivio Istituzionale della Ricerca - Università degli Studi della Campania "Luigi Vanvitelli" (submittedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- artificial intelligence in healthcare, machine learning, clinical research quality, AI literacy, clinical relevance, information leakage, bias-variance trade-off, model evaluation, clinical utility, reporting transparency, open science, data quality, generalizability, clinician-AI collaboration
- MeSH terms
- Machine Learning, Algorithms, Artificial Intelligence, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
