Veridical data science
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Building and expanding on principles of statistics, machine learning, and scientific inquiry, we propose the predictability, computability, and stability (PCS) framework for veridical data science. Our framework, composed of both a workflow and documentation, aims to provide responsible, reliable, reproducible, and transparent results across the data science life cycle. The PCS workflow uses predictability as a reality check and considers the importance of computation in data collection/storage and algorithm design. It augments predictability and computability with an overarching stability principle. Stability expands on statistical uncertainty considerations to assess how human judgment calls impact data results through data and model/algorithm perturbations. As part of the PCS workflow, we develop PCS inference procedures, namely PCS perturbation intervals and PCS hypothesis testing, to investigate the stability of data results relative to problem formulation, data cleaning, modeling decisions, and interpretations. We illustrate PCS inference through neuroscience and genomics projects of our own and others. Moreover, we demonstrate its favorable performance over existing methods in terms of receiver operating characteristic (ROC) curves in high-dimensional, sparse linear model simulations, including a wide range of misspecified models. Finally, we propose PCS documentation based on R Markdown or Jupyter Notebook, with publicly available, reproducible codes and narratives to back up human choices made throughout an analysis. The PCS workflow and documentation are demonstrated in a genomics case study available on Zenodo.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
Bootstrap methods: another look at the jackknife
Efron, B. (1979) · The Annals of Statistics · doi:10.1214/aos/1176344552 · Cited by 17,602 · cited 1×
“For example, jackknife (17–19), bootstrap (20), and cross-validation (6, 7) may be considered appropriate perturbations if the data are deemed approximately independent and identically distributed (i.”
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
A Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643
On computable numbers, with an application to the entscheidungsproblem
Turing, A. M. (1937) · Proceedings of the London Mathematical Society · doi:10.1112/plms/s2-42.1.230 · Cited by 8,283
Strictly Proper Scoring Rules, Prediction, and Estimation
Gneiting, T., & Raftery, A. E. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001437 · Cited by 5,793
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“The limited transparency in reporting such judgment calls has blurred the evidence for many analyses, resulting in more false discoveries than might otherwise occur (2, 3).”
BackgroundStatistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340 · cited 2×
“Leo Breiman championed the essential role of prediction in developing realistic models that yield sound scientific conclusions (6).”
Background
Cited by
Computational approaches streamlining drug discovery
Sadybekov, A. V., & Katritch, V. (2023) · Nature · doi:10.1038/s41586-023-05905-z · Cited by 1,279
Navigating the pitfalls of applying machine learning in genomics
Whalen, S., Schreiber, J., Noble, W. S., & Pollard, K. S. (2022) · Nature Reviews Genetics · doi:10.1038/s41576-021-00434-9 · Cited by 271
Stable reliability diagrams for probabilistic classifiers
Dimitriadis, T., Gneiting, T., & Jordan, A. I. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016191118 · Cited by 84
“drastically distinct reliability diagrams. This is a disconcerting state of aairs for a widely used data analytic tool, and contrary to well-argued recent pleas for reproducibility (8) and stability (9). Similar instabilities under the binning and counting approach have been reported for numerical measures of calibrat…”
ResultInfluentialImproving reproducibility by using high-throughput observational studies with empirical calibration
Schuemie, M. J., Ryan, P. B., Hripcsak, G., Madigan, D., & Suchard, M. A. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2017.0356 · Cited by 74
Cheng, F., Wang, F., Tang, J., Zhou, Y., Fu, Z., Zhang, P., Haines, J. L., Leverenz, J. B., Gan, L., Hu, J., et al. (2024) · Cell Reports Medicine · doi:10.1016/j.xcrm.2023.101379 · Cited by 63
Metadata
- Authors
- Yu, B., & Kumbier, K.
- Venue
- Proceedings of the National Academy of Sciences
- Published
- 2020-02-13
- Volume
- 117
- Issue
- 8
- Pages
- 3920-3929
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- OpenAlex
- W2986518719
- Cited by
- 164
- References
- 19
- FWCI
- 21.57
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Veridical data science. https://app.cochat.ai/papers/10.1073/pnas.1901326117
- The paper
- Yu, B., & Kumbier, K. (2020). Veridical data science. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.1901326117
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.