Aggregate and individual replication probability within an explicit model of the research process.
The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
We study a model of the research process in which the true effect size, the replication jitter due to changes in experimental procedure, and the statistical error of effect size measurement are all normally distributed random variables. Within this model, we analyze the probability of successfully replicating an initial experimental result by obtaining either a statistically significant result in the same direction or any effect in that direction. We analyze both the probability of successfully replicating a particular experimental effect (i.e., the individual replication probability) and the average probability of successful replication across different studies within some research context (i.e., the aggregate replication probability), and we identify the conditions under which the latter can be approximated using the formulas of Killeen (2005a, 2007). We show how both of these probabilities depend on parameters of the research context that would rarely be known in practice. In addition, we show that the statistical uncertainty associated with the size of an initial observed effect would often prevent accurate estimation of the desired individual replication probability even if these research context parameters were known exactly. We conclude that accurate estimates of replication probability are generally unattainable.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis
Hoenig, J. M., & Heisey, D. M. (2001) · The American Statistician · doi:10.1198/000313001300339897 · Cited by 2,046
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824
One Hundred Years of Social Psychology Quantitatively Described
Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003) · Review of General Psychology · doi:10.1037/1089-2680.7.4.331 · Cited by 1,558
Cited by
Performing high‐powered studies efficiently with sequential analyses
Lakens, D. (2014) · European Journal of Social Psychology · doi:10.1002/ejsp.2023 · Cited by 414
Still, C. D., Wood, G. C., Benotti, P., Petrick, A. T., Gabrielsen, J., Strodel, W. E., Ibele, A., Seiler, J., Irving, B. A., Celaya, M. P., et al. (2014) · The Lancet Diabetes & Endocrinology · doi:10.1016/s2213-8587(13)70070-6 · Cited by 350
Lakens, D., & Evers, E. R. K. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528520 · Cited by 266
Rewarding Replications: A Sure and Simple Way to Improve Psychological Science
Koole, S. L., & Lakens, D. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462586 · Cited by 281
Foundations of multicultural psychology: Research to inform effective practice.
Smith, T. B., & Trimble, J. E. (2016) · doi:10.1037/14733-000 · Cited by 132
Metadata
- Authors
- Miller, J., & Schwarz, W.
- Venue
- Psychological Methods
- Published
- 2011-09
- Volume
- 16
- Issue
- 3
- Pages
- 337-360
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W1986038840
- Cited by
- 40
- References
- 3
- FWCI
- 3.38
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.