CoChat
Sign Up
PreprintCited by 12Adversarial Robustness in Machine LearningMachine Learning in HealthcareCancer Diagnosis and Treatment

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology

Open access
MacDonald, Samual, Foley, Helena, Yap, Melvyn, Johnston, Rebecca, Steven, Kaiah, Koufariotis, Lambros, Sharma, Somwya, Wood, Scott, Addala, Venkateswar, Pearson, John, Roosta, Fred, Waddell, Nicola, Kondrashova, Olga, Trzaskowski, Maciej·2022·Scientific Reports
DOI 10.21203/rs.3.rs-1984539/v1PMC PMC10164181

Preprint — published as 10.1038/s41598-023-31126-5 · Paper page

Open this paper in CoChat — free accountOpen PDF

Abstract

Trust and transparency are critical for deploying deep learning (DL) models into the clinic. DL application poses generalisation obstacles since training/development datasets often have different data distributions to clinical/production datasets that can lead to incorrect predictions with underestimated uncertainty. To investigate this pitfall, we benchmarked one pointwise and three approximate Bayesian DL models used to predict cancer of unknown primary with three independent RNA-seq datasets covering 10,968 samples across 57 primary cancer types. Our results highlight simple and scalable Bayesian DL significantly improves the generalisation of uncertainty estimation (e.g., p-value = 0.0013 for calibration). Moreover, we demonstrate Bayesian DL substantially improves accuracy under data distributional shifts when utilising ‘uncertainty thresholding’ by designing a prototypical metric that evaluates the expected (accuracy) loss when deploying models from development to production, which we call the Area between Development and Production curve (ADP). In summary, Bayesian DL is a hopeful avenue of research for generalising uncertainty, which improves performance, transparency, and therefore safety of DL models for deployment in real-world.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.21203/rs.3.rs-1984539/v1

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 72

  1. C. Cao et al., “Deep Learning and Its Applications in Biomedicine,” Genomics Proteomics Bioinformatics, vol. 16, no. 1, pp. 17–32, Feb. 2018, doi: 10.1016/j.gpb.2017.07.003. · DOI
  2. K. A. Tran, O. Kondrashova, A. Bradley, E. D. Williams, J. V. Pearson, and N. Waddell, “Deep learning in cancer diagnosis, prognosis and treatment selection,” Genome Med., vol. 13, no. 1, p. 152, Sep. 2021, doi: 10.1186/s13073-021-00968-x. · DOI · Paper page
  3. M. Wang, Q. Zhang, S. Lam, J. Cai, and R. Yang, “A Review on Application of Deep Learning Algorithms in External Beam Radiotherapy Automated Treatment Planning,” Front. Oncol., vol. 10, 2020, Accessed: Jun. 06, 2022. [Online]. Available: https://www.frontiersin.org/article/10.3389/fonc.2020.580919 · DOI
  4. W. Zhu, L. Xie, J. Han, and X. Guo, “The Application of Deep Learning in Cancer Prognosis Prediction,” Cancers, vol. 12, no. 3, Mar. 2020, doi: 10.3390/cancers12030603. · DOI
  5. P. Schelb et al., “Classification of Cancer at Prostate MRI: Deep Learning versus Clinical PI-RADS Assessment,” Radiology, vol. 293, no. 3, pp. 607–617, Dec. 2019, doi: 10.1148/radiol.2019190938. · DOI
  6. O. Ozdemir, R. Russell, and A. Berlin, “A 3D Probabilistic Deep Learning System for Detection and Diagnosis of Lung Cancer Using Low-Dose CT Scans,” IEEE Trans. Med. Imaging, vol. PP, pp. 1–1, Oct. 2019, doi: 10.1109/TMI.2019.2947595. · DOI
  7. A. Su et al., “A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images,” Npj Precis. Oncol., vol. 6, no. 1, Art. no. 1, Mar. 2022, doi: 10.1038/s41698-022-00252-0. · DOI
  8. W. Jiao et al., “A deep learning system accurately classifies primary and metastatic cancers using passenger mutation patterns,” Nat. Commun., vol. 11, no. 1, Art. no. 1, Feb. 2020, doi: 10.1038/s41467-019-13825-8. · DOI
  9. Z. K. Tuong et al., “Resolving the immune landscape of human prostate at a single-cell level in health and cancer,” Cell Rep., vol. 37, no. 12, p. 110132, Dec. 2021, doi: 10.1016/j.celrep.2021.110132. · DOI
  10. M. Yap et al., “Verifying explainability of a deep learning tissue classifier trained on RNA-seq data,” Sci. Rep., vol. 11, no. 1, Art. no. 1, Jan. 2021, doi: 10.1038/s41598-021-81773-9. · DOI
  11. A. Gayoso et al., “A Python library for probabilistic analysis of single-cell omics data,” Nat. Biotechnol., vol. 40, no. 2, Art. no. 2, Feb. 2022, doi: 10.1038/s41587-021-01206-w. · DOI · Paper page
  12. M. D. Luecken et al., “A sandbox for prediction and integration of DNA, RNA, and proteins in single cells,” presented at the Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), Aug. 2021. Accessed: Jun. 06, 2022. [Online]. Available: https://openreview.net/forum?id=gN35BGa1Rt
  13. C. M. Park and J. H. Lee, “Deep Learning for Lung Cancer Nodal Staging and Real-World Clinical Practice,” Radiology, vol. 302, no. 1, pp. 212–213, Jan. 2022, doi: 10.1148/radiol.2021211981. · DOI
  14. J. Weberpals et al., “Deep Learning-based Propensity Scores for Confounding Control in Comparative Effectiveness Research: A Large-scale, Real-world Data Study,” Epidemiol. Camb. Mass, vol. 32, no. 3, pp. 378–388, May 2021, doi: 10.1097/EDE.0000000000001338. · DOI
  15. S. MacDonald, S. Kaiah, and M. Trzaskowski, “Interpretable AI in Healthcare: Enhancing Fairness, Safety, and Trust,” in Artificial Intelligence in Medicine: Applications, Limitations and Future Directions, Springer, Singapore, 2022, pp. 241–258. · DOI
  16. C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead,” Nat. Mach. Intell., vol. 1, no. 5, Art. no. 5, May 2019, doi: 10.1038/s42256-019-0048-x. · DOI · Paper page
  17. Y. Gal, “Uncertainty in Deep Learning,” PhD, University of Cambridge, 2016.
  18. J. Gawlikowski et al., “A Survey of Uncertainty in Deep Neural Networks,” arXiv, arXiv:2107.03342, Jan. 2022. doi: 10.48550/arXiv.2107.03342. · DOI · Paper page
  19. A. Barragán-Montero et al., “Towards a safe and efficient clinical implementation of machine learning in radiation oncology by exploring model interpretability, explainability and data-model dependency,” Phys. Med. Ampmathsemicolon Biol., vol. 67, no. 11, p. 11TR01, May 2022, doi: 10.1088/1361-6560/ac678a. · DOI
  20. A. Kristiadi, M. Hein, and P. Hennig, “Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks,” arXiv, arXiv:2002.10118, Jul. 2020. doi: 10.48550/arXiv.2002.10118. · DOI
  21. M. Minderer et al., “Revisiting the Calibration of Modern Neural Networks,” in Advances in Neural Information Processing Systems, 2021, vol. 34, pp. 15682–15694. Accessed: Jun. 06, 2022. [Online]. Available: https://proceedings.neurips.cc/paper/2021/hash/8420d359404024567b5aefda1231af24-Abstract.html
  22. Y. Ovadia et al., “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift,” arXiv, arXiv:1906.02530, Dec. 2019. doi: 10.48550/arXiv.1906.02530. · DOI · Paper page
  23. R. M. French, “Catastrophic forgetting in connectionist networks,” Trends Cogn. Sci., vol. 3, no. 4, pp. 128–135, Apr. 1999, doi: 10.1016/S1364-6613(99)01294-2. · DOI · Paper page
  24. S. Gupta et al., “Addressing catastrophic forgetting for medical domain expansion,” arXiv, arXiv:2103.13511, Mar. 2021. doi: 10.48550/arXiv.2103.13511. · DOI
  25. C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On Calibration of Modern Neural Networks,” arXiv, arXiv:1706.04599, Aug. 2017. doi: 10.48550/arXiv.1706.04599. · DOI · Paper page
  26. M. E. Khan and H. Rue, “The Bayesian Learning Rule,” arXiv, arXiv:2107.04562, Mar. 2022. doi: 10.48550/arXiv.2107.04562. · DOI
  27. A. G. Wilson and P. Izmailov, “Bayesian Deep Learning and a Probabilistic Perspective of Generalization,” Feb. 2020, doi: 10.48550/arXiv.2002.08791. · DOI
  28. M. Divate, A. Tyagi, D. J. Richard, P. A. Prasad, H. Gowda, and S. H. Nagaraj, “Deep Learning-Based Pan-Cancer Classification Model Reveals Tissue-of-Origin Specific Gene Expression Signatures,” Cancers, vol. 14, no. 5, Art. no. 5, Jan. 2022, doi: 10.3390/cancers14051185. · DOI · Paper page
  29. J. K. Grewal et al., “Application of a Neural Network Whole Transcriptome–Based Pan-Cancer Method for Diagnosis of Primary and Metastatic Cancers,” JAMA Netw. Open, vol. 2, no. 4, p. e192597, Apr. 2019, doi: 10.1001/jamanetworkopen.2019.2597. · DOI · Paper page
  30. Y. Zhao et al., “CUP-AI-Dx: A tool for inferring cancer tissue of origin and molecular subtype using RNA gene-expression data and artificial intelligence,” EBioMedicine, vol. 61, p. 103030, Nov. 2020, doi: 10.1016/j.ebiom.2020.103030. · DOI
  31. K. Tomczak, P. Czerwińska, and M. Wiznerowicz, “The Cancer Genome Atlas (TCGA): an immeasurable source of knowledge,” Contemp. Oncol. Poznan Pol., vol. 19, no. 1A, pp. A68-77, 2015, doi: 10.5114/wo.2014.47136. · DOI
  32. K. A. Hoadley et al., “Cell-of-Origin Patterns Dominate the Molecular Classification of 10,000 Tumors from 33 Types of Cancer,” Cell, vol. 173, no. 2, pp. 291–304.e6, Apr. 2018, doi: 10.1016/j.cell.2018.03.022. · DOI · Paper page
  33. D. R. Robinson et al., “Integrative clinical genomics of metastatic cancer,” Nature, vol. 548, no. 7667, Art. no. 7667, Aug. 2017, doi: 10.1038/nature23306. · DOI · Paper page
  34. S. Akgül et al., “Intratumoural Heterogeneity Underlies Distinct Therapy Responses and Treatment Resistance in Glioblastoma,” Cancers, vol. 11, no. 2, Art. no. 2, Feb. 2019, doi: 10.3390/cancers11020190. · DOI · Paper page
  35. L. G. Aoude et al., “Radiomics Biomarkers Correlate with CD8 Expression and Predict Immune Signatures in Melanoma Patients,” Mol. Cancer Res., vol. 19, no. 6, pp. 950–956, Jun. 2021, doi: 10.1158/1541-7786.MCR-20-1038. · DOI
  36. P. Bailey et al., “Genomic analyses identify molecular subtypes of pancreatic cancer,” Nature, vol. 531, no. 7592, Art. no. 7592, Mar. 2016, doi: 10.1038/nature16965. · DOI · Paper page
  37. N. K. Hayward et al., “Whole-genome landscapes of major melanoma subtypes,” Nature, vol. 545, no. 7653, Art. no. 7653, May 2017, doi: 10.1038/nature22071. · DOI
  38. J. H. Lee et al., “Transcriptional downregulation of MHC class I and melanoma de- differentiation in resistance to PD-1 inhibition,” Nat. Commun., vol. 11, no. 1, Art. no. 1, Apr. 2020, doi: 10.1038/s41467-020-15726-7. · DOI
  39. F. Newell et al., “Multiomic profiling of checkpoint inhibitor-treated melanoma: Identifying predictors of response and resistance, and markers of biological discordance,” Cancer Cell, vol. 40, no. 1, pp. 88–102.e7, Jan. 2022, doi: 10.1016/j.ccell.2021.11.012. · DOI
  40. F. Newell et al., “Whole-genome sequencing of acral melanoma reveals genomic complexity and diversity,” Nat. Commun., vol. 11, no. 1, p. 5259, Oct. 2020, doi: 10.1038/s41467-020-18988-3. · DOI
  41. A.-M. Patch et al., “Whole–genome characterization of chemoresistant ovarian cancer,” Nature, vol. 521, no. 7553, Art. no. 7553, May 2015, doi: 10.1038/nature14410. · DOI
  42. A. Scarpa et al., “Whole-genome landscape of pancreatic neuroendocrine tumours,” Nature, vol. 543, no. 7643, Art. no. 7643, Mar. 2017, doi: 10.1038/nature21063. · DOI · Paper page
  43. Y. Gal and Z. Ghahramani, “Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning,” arXiv, arXiv:1506.02142, Oct. 2016. doi: 10.48550/arXiv.1506.02142. · DOI · Paper page
  44. J. Z. Liu, Z. Lin, S. Padhy, D. Tran, T. Bedrax-Weiss, and B. Lakshminarayanan, “Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness,” arXiv, arXiv:2006.10108, Oct. 2020. doi: 10.48550/arXiv.2006.10108. · DOI · Paper page
  45. J. van Amersfoort, L. Smith, A. Jesson, O. Key, and Y. Gal, “On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty,” arXiv, arXiv:2102.11409, Mar. 2022. doi: 10.48550/arXiv.2102.11409. · DOI
  46. J. van Amersfoort, L. Smith, Y. W. Teh, and Y. Gal, “Uncertainty Estimation Using a Single Deep Deterministic Neural Network,” arXiv, arXiv:2003.02037, Jun. 2020. doi: 10.48550/arXiv.2003.02037. · DOI
  47. A. Malinin et al., “Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks,” arXiv, arXiv:2107.07455, Feb. 2022. doi: 10.48550/arXiv.2107.07455. · DOI · Paper page
  48. P. Izmailov, P. Nicholson, S. Lotfi, and A. G. Wilson, “Dangers of Bayesian Model Averaging under Covariate Shift,” in Advances in Neural Information Processing Systems, 2021, vol. 34, pp. 3309–3322. Accessed: Jun. 06, 2022. [Online]. Available: https://proceedings.neurips.cc/paper/2021/hash/1ab60b5e8bd4eac8a7537abb5936aadc-Abstract.html
  49. J. Mukhoti, P. Stenetorp, and Y. Gal, “On the Importance of Strong Baselines in Bayesian Deep Learning,” arXiv, arXiv:1811.09385, Nov. 2018. doi: 10.48550/arXiv.1811.09385. · DOI
  50. Kevin P. Murphy, “Inference algorithms: an overview,” in Probabilistic Machine Learning: Advanced Topics (draft), MIT Press, 2022, p. 319. [Online]. Available: probml.ai
  51. M. Abdar et al., “A review of uncertainty quantification in deep learning: Techniques, applications and challenges,” Inf. Fusion, vol. 76, pp. 243–297, Dec. 2021, doi: 10.1016/j.inffus.2021.05.008. · DOI · Paper page
  52. E. Hüllermeier and W. Waegeman, “Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods,” Mach. Learn., vol. 110, no. 3, pp. 457–506, Mar. 2021, doi: 10.1007/s10994-021-05946-3. · DOI · Paper page
  53. A. Jesson, S. Mindermann, Y. Gal, and U. Shalit, “Quantifying Ignorance in Individual-Level Causal-Effect Estimates under Hidden Confounding,” in Proceedings of the 38th International Conference on Machine Learning, Jul. 2021, pp. 4829–4838. Accessed: Jun. 29, 2022. [Online]. Available: https://proceedings.mlr.press/v139/jesson21a.html
  54. A. S. Sambyal, N. C. Krishnan, and D. R. Bathula, “Towards Reducing Aleatoric Uncertainty for Medical Imaging Tasks.” arXiv, May 08, 2022. doi: 10.48550/arXiv.2110.11012. · DOI
  55. S. W. Ober, C. E. Rasmussen, and M. van der Wilk, “The Promises and Pitfalls of Deep Kernel Learning,” arXiv, arXiv:2102.12108, Jul. 2021. doi: 10.48550/arXiv.2102.12108. · DOI
  56. M. M. Bronstein, J. Bruna, T. Cohen, and P. Veličković, “Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges.” arXiv, May 02, 2021. doi: 10.48550/arXiv.2104.13478. · DOI
  57. C. Shorten and T. M. Khoshgoftaar, “A survey on Image Data Augmentation for Deep Learning,” J. Big Data, vol. 6, no. 1, p. 60, Jul. 2019, doi: 10.1186/s40537-019-0197-0. · DOI · Paper page
  58. J. Peters, D. Janzing, and B. Schölkopf, Elements of Causal Inference: Foundations and Learning Algorithms. Cambridge, MA, USA: MIT Press, 2017.
  59. B. Schölkopf et al., “Toward Causal Representation Learning,” Proc. IEEE, vol. 109, no. 5, pp. 612–634, May 2021, doi: 10.1109/JPROC.2021.3058954. · DOI · Paper page
  60. K. Xia, K.-Z. Lee, Y. Bengio, and E. Bareinboim, “The Causal-Neural Connection: Expressiveness, Learnability, and Inference,” in Advances in Neural Information Processing Systems, 2021, vol. 34, pp. 10823–10836. Accessed: Jun. 29, 2022. [Online]. Available: https://proceedings.neurips.cc/paper/2021/hash/5989add1703e4b0480f75e2390739f34-Abstract.html
  61. A. D’Amour et al., “Underspecification Presents Challenges for Credibility in Modern Machine Learning.” arXiv, Nov. 24, 2020. doi: 10.48550/arXiv.2011.03395. · DOI · Paper page
  62. A. Kryshtafovych, T. Schwede, M. Topf, K. Fidelis, and J. Moult, “Critical assessment of methods of protein structure prediction (CASP)—Round XIV,” Proteins Struct. Funct. Bioinforma., vol. 89, no. 12, pp. 1607–1617, 2021, doi: 10.1002/prot.26237. · DOI · Paper page
  63. D. Misra, “Mish: A Self Regularized Non-Monotonic Activation Function,” arXiv, arXiv:1908.08681, Aug. 2020. doi: 10.48550/arXiv.1908.08681. · DOI · Paper page
  64. S. Ioffe and C. Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” arXiv, arXiv:1502.03167, Mar. 2015. doi: 10.48550/arXiv.1502.03167. · DOI · Paper page
  65. J. Behrmann, W. Grathwohl, R. T. Q. Chen, D. Duvenaud, and J.-H. Jacobsen, “Invertible Residual Networks,” arXiv, arXiv:1811.00995, May 2019. doi: 10.48550/arXiv.1811.00995. · DOI
  66. K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” arXiv, arXiv:1512.03385, Dec. 2015. doi: 10.48550/arXiv.1512.03385. · DOI · Paper page
  67. F. Farnia, J. M. Zhang, and D. Tse, “Generalizable Adversarial Training via Spectral Normalization,” arXiv, arXiv:1811.07457, Nov. 2018. doi: 10.48550/arXiv.1811.07457. · DOI
  68. S. Fort, H. Hu, and B. Lakshminarayanan, “Deep Ensembles: A Loss Landscape Perspective,” arXiv, arXiv:1912.02757, Jun. 2020. doi: 10.48550/arXiv.1912.02757. · DOI
  69. P. Izmailov, S. Vikram, M. D. Hoffman, and A. G. Wilson, “What Are Bayesian Neural Network Posteriors Really Like?,” arXiv, arXiv:2104.14421, Apr. 2021. doi: 10.48550/arXiv.2104.14421. · DOI
  70. F. D’ Angelo and V. Fortuin, “Repulsive Deep Ensembles are Bayesian,” in Advances in Neural Information Processing Systems, 2021, vol. 34, pp. 3451–3465. Accessed: Jun. 30, 2022. [Online]. Available: https://proceedings.neurips.cc/paper/2021/hash/1c63926ebcabda26b5cdb31b5cc91efb-Abstract.html
  71. J. Mukhoti, A. Kirsch, J. van Amersfoort, P. H. S. Torr, and Y. Gal, “Deep Deterministic Uncertainty: A Simple Baseline,” arXiv, arXiv:2102.11582, Jan. 2022. doi: 10.48550/arXiv.2102.11582. · DOI
  72. K. Zhang, B. Schölkopf, K. Muandet, and Z. Wang, “Domain Adaptation under Target and Conditional Shift,” in Proceedings of the 30th International Conference on Machine Learning, May 2013, pp. 819–827. Accessed: Jun. 30, 2022. [Online]. Available: https://proceedings.mlr.press/v28/zhang13d.html

Metadata

Authors
MacDonald, S., Foley, H., Yap, M., Johnston, R., Steven, K., Koufariotis, L., Sharma, S., Wood, S., Addala, V., Pearson, J., Roosta, F., Waddell, N., Kondrashova, O., & Trzaskowski, M.
Affiliations
  • The University of Queensland (AU)
  • Max Kelsen
  • QIMR Berghofer Medical Research Institute (AU)
  • QIMR Berghofer Medical Research Institute, Medlab Pathology
Corresponding author
  • Kondrashova, Olga
  • Trzaskowski, Maciej
ORCID
Venue
Scientific Reports
Publisher
Research Square Platform LLC
Published
2022-08-29
Volume
13
Issue
1
Pages
7395-7395
ISSN
2045-2322 (ISSN-L)
DOI
10.21203/rs.3.rs-1984539/v1
PMC
PMC10164181
OpenAlex
W4372354552
Cited by
12
References
1
Citations by year
2026: 5 · 2025: 3 · 2024: 2 · 2023: 2
FWCI
1.13
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
  • Research Square (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • QIMR Berghofer Medical Research Institute
  • Cooperative Research Centres, Australian Government Department of Industry (CRCPFIVE000176)
  • Medical Research Council
  • Australian Research Council (IC200100022)
  • National Health and Medical Research Council (2008631, 1139071)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Oncology › Cancer Diagnosis and Treatment
Keywords
uncertainty estimation, cancer of unknown primary, RNA-seq, model generalization
MeSH terms
Deep Learning, Bayes Theorem, Medical Oncology, Reproducibility of Results, Uncertainty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review