The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Continuously Cumulating Meta-Analysis and Replicability
Open accessAbstract
The current crisis in scientific psychology about whether our findings are irreproducible was presaged years ago by Tversky and Kahneman (1971), who noted that even sophisticated researchers believe in the fallacious Law of Small Numbers—erroneous intuitions about how imprecisely sample data reflect population phenomena. Combined with the low power of most current work, this often leads to the use of misleading criteria about whether an effect has replicated. Rosenthal (1990) suggested more appropriate criteria, here labeled the continuously cumulating meta-analytic (CCMA) approach. For example, a CCMA analysis on a replication attempt that does not reach significance might nonetheless provide more, not less, evidence that the effect is real. Alternatively, measures of heterogeneity might show that two studies that differ in whether they are significant might have only trivially different effect sizes. We present a nontechnical introduction to the CCMA framework (referencing relevant software), and then explain how it can be used to address aspects of replicability or more generally to assess quantitative evidence from numerous studies. We then present some examples and simulation results using the CCMA approach that show how the combination of evidence can yield improved results over the consideration of single studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
(2000) · Cited by 9,198
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Self-citationSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Assessing heterogeneity in meta-analysis: Q statistic or I² index?
Huedo-Medina, T. B., Sánchez-Meca, J., Marín-Martínez, F., & Botella, J. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.2.193 · Cited by 4,256
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Funnel plots for detecting bias in meta-analysis
Sterne, J. A. C., & Egger, M. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00377-8 · Cited by 3,779
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246
Bargh, J. A., Chen, M., & Burrows, L. (1996) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.71.2.230 · Cited by 2,736
Bargh, J. A., Chen, M. I.-C., & Burrows, L. (1996) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.71.2.230 · Cited by 2,636
Ellis, P. D. (2010) · doi:10.1017/cbo9780511761676 · Cited by 2,461
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033 · cited 3×
“…size (Cohen, 1988; r = 0.243), and is in the range of effect sizes typically found in published psychological research by recent reviews (Anderson, Lindsay, & Bushman, 1999; Hall, 1998; Lipsey & Wilson, 1993; Meyer et al., 2001; Richard, Bond, & Stokes-Zoota, 2003; Tett, Meyer, & Roese, 1994).”
BackgroundInfluential(2011) · Cited by 2,025
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824
One Hundred Years of Social Psychology Quantitatively Described
Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003) · Review of General Psychology · doi:10.1037/1089-2680.7.4.331 · Cited by 1,558 · cited 1×
“Readers can find formulas and examples for calculating each of these by going to Rosenthal and Rubin (1982) and Higgins et al.”
MethodPsychological testing and psychological assessment: A review of evidence and issues.
Meyer, G. J., Finn, S. E., Eyde, L. D., Kay, G. G., Moreland, K. L., Dies, R. R., Eisman, E. J., Kubiszyn, T. W., & Reed, G. M. (2001) · American Psychologist · doi:10.1037//0003-066x.56.2.128 · Cited by 1,469 · cited 4×
“243), and is in the range of effect sizes typically found in published psychological research by recent reviews (Anderson, Lindsay, & Bushman, 1999; Hall, 1998; Lipsey & Wilson, 1993; Meyer et al., 2001; Richard, Bond, & Stokes-Zoota, 2003; Tett, Meyer, & Roese, 1994).”
BackgroundInfluentialReview papers : The statistical basis of meta-analysis
Fleiss, J. (1993) · Statistical Methods in Medical Research · doi:10.1177/096228029300200202 · Cited by 1,281
Uncertainty in heterogeneity estimates in meta-analyses
Ioannidis, J. P. A., Patsopoulos, N. A., & Evangelou, E. (2007) · BMJ · doi:10.1136/bmj.39343.408449.80 · Cited by 1,254
Investigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
Patsopoulos, N. A., Evangelou, E., & Ioannidis, J. P. (2008) · International Journal of Epidemiology · doi:10.1093/ije/dyn065 · Cited by 1,138
Estimation of effect size from a series of independent experiments.
Hedges, L. V. (1982) · Psychological Bulletin · doi:10.1037//0033-2909.92.2.490 · Cited by 1,062
Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests
Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406923 · Cited by 945
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800
Understanding The New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis
Cumming, G. (2013) · Routledge eBooks · doi:10.4324/9780203807002 · Cited by 790
Statistical procedures and the justification of knowledge in psychological science.
Rosnow, R. L., & Rosenthal, R. (1989) · American Psychologist · doi:10.1037/0003-066x.44.10.1276 · Cited by 777
Self-citationReplications in Psychology Research: How Often Do They Really Occur?
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460688 · Cited by 750
Do studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748
Small Telescopes: Detectability and the Evaluation of Replication Results
Simonsohn, U. (2015) · Psychological Science · doi:10.1177/0956797614567341 · Cited by 742
The fallacy of the null-hypothesis significance test.
Rozeboom, W. W. (1960) · Psychological Bulletin · doi:10.1037/h0042040 · Cited by 729
Is the Replicability Crisis Overblown? Three Arguments Examined
Pashler, H., & Harris, C. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463401 · Cited by 697
Behavioral Priming: It’s all in the Mind, but whose Mind?
Doyen, S., Klein, O., Pichon, C.-L., & Cleeremans, A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0029081 · Cited by 670
Statistical procedures and the justification of knowledge in psychological science.
Rosnow, R. L., & Rosenthal, R. (1992) · Methodological issues & strategies in clinical research. · doi:10.1037/10109-027 · Cited by 652
Self-citationAn Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science
Open Science Collaboration (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462588 · Cited by 614
Thompson, B. (2002) · Educational Researcher · doi:10.3102/0013189x031003025 · Cited by 538
Cumming, G., & Finch, S. (2001) · Educational and Psychological Measurement · doi:10.1177/0013164401614002 · Cited by 528
Research in the Psychological Laboratory: Truth or Triviality?
Anderson, C. A., Lindsay, J. J., & Bushman, B. J. (1999) · Current Directions in Psychological Science · doi:10.1111/1467-8721.00002 · Cited by 505
Approaches to heterogeneity in meta‐analysis
Petitti, D. B. (2001) · Statistics in Medicine · doi:10.1002/sim.1091 · Cited by 464
The ironic effect of significant results on the credibility of multiple-study articles.
Schimmack, U. (2012) · Psychological Methods · doi:10.1037/a0029487 · Cited by 455
Cumming, G. (2008) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2008.00079.x · Cited by 451
A comparison of statistical methods for combining event rates from clinical trials
Berlin, J. A., Laird, N. M., Sacks, H. S., & Chalmers, T. C. (1989) · Statistics in Medicine · doi:10.1002/sim.4780080202 · Cited by 420
Cooper, H. M. (1979) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.37.1.131 · Cited by 412
Comparing effect sizes of independent studies.
Rosenthal, R., & Rubin, D. B. (1982) · Psychological Bulletin · doi:10.1037//0033-2909.92.2.500 · Cited by 406
Self-citationGiner-Sorolla, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612457576 · Cited by 274 · cited 2×
“, those replication studies that have been attempted fail to produce supportive findings) or that journals are biased against replication studies whatever their result (Giner-Sorolla, 2012; Neuliep & Crandall, 1990, 1993; Rosenthal, 1979).”
ResultLakens, D., & Evers, E. R. K. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528520 · Cited by 266 · cited 1×
“The call for higher power is certainly not a new one (see, e.g., Cohen, 1962; Lakens & Evers, 2014, this issue; Sedlmeier & Gigerenzer, 1989), however it is worth repeating.”
BackgroundReplication in behavioral research.
Rosenthal, R. (1990) · Journal of social behavior and personality · Cited by 214
Self-citationExpectations for Replications: Are Yours Realistic?
Stanley, D. J., & Spence, J. R. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528518 · Cited by 209
Too good to be true: Publication bias in two prominent studies from experimental psychology
Francis, G. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0227-9 · Cited by 206
Safeguard Power as a Protection Against Imprecise Power Estimates
Perugini, M., Gallucci, M., & Costantini, G. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528519 · Cited by 198
Maner, J. K. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528215 · Cited by 168
Sample size in psychological research over the past 30 years
Marszalek, J. M., Barber, C., Kohlhart, J., & Cooper, B. H. (2011) · Perceptual and Motor Skills · doi:10.2466/03.11.pms.112.2.331-348 · Cited by 167
Replication in prevention science
Valentine, J. C., Biglan, A., Boruch, R. F., Castro, F. G., Collins, L. M., Flay, B. R., Kellam, S., Mościcki, E. K., & Schinke, S. P. (2011) · Prevention Science · doi:10.1007/s11121-011-0217-6 · Cited by 166
Two Failures to Replicate High-Performance-Goal Priming Effects
Harris, C. R., Coburn, N., Rohrer, D., & Pashler, H. (2013) · PLoS ONE · doi:10.1371/journal.pone.0072467 · Cited by 159
Editorial bias against replication research.
Neuliep, J. W., & Crandall, R. (1990) · Journal of social behavior and personality · Cited by 139
Prestige, Suggestion, and Attitudes
Lorge, I. D., & Curtiss, C. (1936) · The Journal of Social Psychology · doi:10.1080/00224545.1936.9919891 · Cited by 118
Reviewer bias against replication research.
Neuliep, J. W., & Crandall, R. (1993) · Journal of social behavior and personality · Cited by 113
Carpenter, S. (2012) · Science · doi:10.1126/science.335.6076.1558 · Cited by 102
Comparing significance levels of independent studies.
Rosenthal, R., & Rubin, D. B. (1979) · Psychological Bulletin · doi:10.1037//0033-2909.86.5.1165 · Cited by 99
Self-citationData from Investigating Variation in Replicability: A “Many Labs” Replication Project
(2014) · Journal of Open Psychology Data · doi:10.5334/jopd.ad · Cited by 86
How Big Are Nonverbal Sex Differences? The Case of Smiling and Sensitivity to Nonverbal Cues
Hall, J. A. (1998) · Psychology Press eBooks · doi:10.4324/9781315805870-7 · Cited by 69
Applications of meta-analysis.
Hunter, J. E., & Hirsh, H. R. (1987) · Cited by 65
Evaluating Replication ResultsPreprint
Simonsohn, U. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2259879 · Cited by 38
P-Curve and Effect Size: Correcting for Publication Bias Using Only Significant ResultsPreprint
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2377290 · Cited by 22
Lakens, D. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2381936 · Cited by 13
Bloch, H. (1994) · Review of Industrial Organization · doi:10.1007/bf01024220 · Cited by 12
Applications of meta-analysis: 1987-1992
Tett, R. P., Meyer, J., & Roese, N. J. (1994) · Cited by 8
Cited by
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102803 · Cited by 2,550
Mini Meta‐Analysis of Your Own Studies: Some Arguments on Why and a Primer on How
Goh, J. X., Hall, J. A., & Rosenthal, R. (2016) · Social and Personality Psychology Compass · doi:10.1111/spc3.12267 · Cited by 994
Self-citationPsychology, Science, and Knowledge Construction: Broadening Perspectives from the Replication Crisis
Shrout, P. E., & Rodgers, J. L. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011845 · Cited by 725
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548
Metadata
- Authors
- Braver, S. L., Thoemmes, F. J., & Rosenthal, R.
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2014-05
- Volume
- 9
- Issue
- 3
- Pages
- 333-342
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2146380848
- Cited by
- 297
- References
- 71
- FWCI
- 32.67
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.