Promoting Replicability in Developmental Research Through Meta-analyses: Insights From Language Acquisition Research
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Previous work suggests that key factors for replicability, a necessary feature for theory building, include statistical power and appropriate research planning. These factors are examined by analyzing a collection of 12 standardized meta-analyses on language development between birth and 5 years. With a median effect size of Cohen's d = .45 and typical sample size of 18 participants, most research is underpowered (range = 6%–99%; median = 44%); and calculating power based on seminal publications is not a suitable strategy. Method choice can be improved, as shown in analyses on exclusion rates and effect size as a function of method. The article ends with a discussion on how to increase replicability in both language acquisition studies specifically and developmental research more generally. The title for this Special Section is Meta-analysis and Individual Participant Data Synthesis in Child Development, edited by Glenn I. Roisman and Marinus H. van IJzendoorn
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 5×
“Small sample sizes, and the ensuing lack of power, are a major risk factor for low replicability (e.g., Button et al., 2013).”
InfluentialSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Morris, S. B., & DeShon, R. P. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.1.105 · Cited by 2,502
Meta-analysis of experiments with matched groups or repeated measures designs
Dunlap, W. P., Cortina, J. M., Vaslow, J. B., & Burke, M. J. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.170 · Cited by 1,693 · cited 1×
“Effect size was calculated when possible from means and standard deviations across designs with the appropriate formulae (Dunlap, Cortina, Vaslow, & Burke, 1996; Lipsey & Wilson, 2001; Morris & DeShon, 2002; Viechtbauer, 2010).”
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Erroneous analyses of interactions in neuroscience: A problem of significance
Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011) · Nature Neuroscience · doi:10.1038/nn.2886 · Cited by 864
The natural selection of bad science
Smaldino, P. E., & McElreath, R. (2016) · Royal Society Open Science · doi:10.1098/rsos.160384 · Cited by 860 · cited 1×
“publishability has been suggested to lead to practices that undermine the quality and reliability of data (Ioannidis, 2005; Smaldino & McElreath, 2016).”
Fixed effects and variance components estimation in three-level meta-analysis
Konstantopoulos, S. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.35 · Cited by 521
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453
Cited by
Sensitivity Analysis for Publication Bias in Meta-Analyses
Mathur, M. B., & VanderWeele, T. J. (2020) · Journal of the Royal Statistical Society Series C: Applied Statistics · doi:10.1111/rssc.12440 · Cited by 354
Quantifying Sources of Variability in Infancy Research Using the Infant-Directed-Speech Preference
The ManyBabies Consortium, Frank, M. C., Alcock, K. J., Arias-Trejo, N., Aschersleben, G., Baldwin, D., Barbu, S., Bergelson, E., Bergmann, C., Black, A. K., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919900809 · Cited by 337
Self-citationHow to conduct a meta-analysis in eight steps: A practical guide
Hansen, C., Steinmetz, H., & Block, J. (2022) · Management Review Quarterly · doi:10.1007/s11301-021-00247-4 · Cited by 218
Transparency and reproducibility of meta-analyses in psychology: A meta-review
Polanin, J. R., Hennessy, E. A., & Tsuji, S. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620906416 · Cited by 100
“Recent technological advances make it unprecedentedly easy to share complete meta-analytic information online, as is demonstrated by the MetaLab initiative (www.metalab.stanford.edu; Bergmann et al., 2018), a website designed for accumulating the meta-analytic raw data of meta-analyses in developmental psychology.”
BackgroundSelf-citation
Metadata
- Authors
- Bergmann, C., Tsuji, S., Piccinini, P. E., Lewis, M. L., Braginsky, M., Frank, M. C., & Cristia, A.
- Venue
- Child Development
- Publisher
- Oxford University Press (OUP)
- Published
- 2018-11-01
- Volume
- 89
- Issue
- 6
- Pages
- 1996-2009
- ISSN
- 0009-3920 (ISSN-L)
- 1467-8624
- OpenAlex
- W2772732614
- Cited by
- 160
- References
- 13
- FWCI
- 13.87
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.