Sample Size in Psychological Research over the Past 30 Years
The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
The American Psychological Association (APA) Task Force on Statistical Inference was formed in 1996 in response to a growing body of research demonstrating methodological issues that threatened the credibility of psychological research, and made recommendations to address them. One issue was the small, even dramatically inadequate, size of samples used in studies published by leading journals. The present study assessed the progress made since the Task Force's final report in 1999. Sample sizes reported in four leading APA journals in 1955, 1977, 1995, and 2006 were compared using nonparametric statistics, while data from the last two waves were fit to a hierarchical generalized linear growth model for more in-depth analysis. Overall, results indicate that the recommendations for increasing sample sizes have not been integrated in core psychological research, although results slightly vary by field. This and other implications are discussed in the context of current methodological critique and practice.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Third, our sample is 50 times larger than the average sample of studies published in high-impact psychology journals (30, 31).”
From Cognitive and noncognitive predictors of success (2019) · cites this paper as [31]
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Answering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225 · cited 1×
“Agreement was measured with Krippendorff’s alpha, α K , using the KALPHA SPSS (now PASW) macro (Hayes & Krippendorff, 2007), treating sample size as a ratio level measurement and performing 1,000 bootstrapping samples.”
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824 · cited 5×
“(1981) parallel the surveys of observed power reported in the literature (Cohen, 1962; Sedlmeier & Gigerenzer, 1989; Rossi, 1990; Maddock & Ros-si, 2001; Maxwell, 2004), and even overlap in specific journals (i.e., Journal of Abnormal Psychology ).”
InfluentialNonparametric statistics for non-statisticians: a step-by-step approach
(2010) · Choice Reviews Online · doi:10.5860/choice.47-3859 · Cited by 1,021
Nonparametric Statistics for Non‐Statisticians: A Step‐by‐Step Approach
Corder, G. W., & Foreman, D. I. (2009) · Wiley eBooks · doi:10.1002/9781118165881 · Cited by 954
Do studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748 · cited 3×
“(1981) parallel the surveys of observed power reported in the literature (Cohen, 1962; Sedlmeier & Gigerenzer, 1989; Rossi, 1990; Maddock & Ros-si, 2001; Maxwell, 2004), and even overlap in specific journals (i.e., Journal of Abnormal Psychology ).”
Maxwell, S. E. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.2.147 · Cited by 654 · cited 8×
“Maxwell (2004) also suggested that editors afford studies with non - significant results with as much chance of publication as studies with sig - nificant results, so that statistical significance is de-emphasized in favor of power and precision.”
InfluentialReporting standards for research in psychology: why do we need them? What might they be?
APA Publications and Communications Board Working Group on Journal Article Reporting Standards (2008) · American Psychologist · doi:10.1037/0003-066x.63.9.839 · Cited by 571
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2019) · CHANCE · doi:10.1080/09332480.2019.1579573 · Cited by 463
The epistemology of mathematical and statistical modeling: A quiet methodological revolution.
Rodgers, J. L. (2010) · American Psychologist · doi:10.1037/a0018326 · Cited by 402 · cited 3×
“This change is in line with the shift from NHSTs to statistical modeling that occurred in the 1990s and 2000s (Rodgers, 2010).”
Research Methods in Psychology
Basavanna, M. (2015) · Psychology for Nurses · doi:10.5005/jp/books/12408_3 · Cited by 399
Statistical power of psychological research: What have we gained in 20 years?
Rossi, J. S. (1990) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.58.5.646 · Cited by 323 · cited 3×
“(1981) parallel the surveys of observed power reported in the literature (Cohen, 1962; Sedlmeier & Gigerenzer, 1989; Rossi, 1990; Maddock & Ros-si, 2001; Maxwell, 2004), and even overlap in specific journals (i.e., Journal of Abnormal Psychology ).”
Educational and psychological measurement
(1982) · Serials Review · doi:10.1016/0098-7913(82)90053-3 · Cited by 186
Research Methods in Psychology
Elmes, D. G., Kantowitz, B. H., & Roediger, H. L., III (1985) · Cited by 159
Dukes, W. F. (1965) · Psychological Bulletin · doi:10.1037/h0021964 · Cited by 128
Fidler, F. (2002) · Educational and Psychological Measurement · doi:10.1177/001316402236876 · Cited by 102
The eclipse of the state mental hospital: policy, stigma, and organization
(1996) · Choice Reviews Online · doi:10.5860/choice.34-1845 · Cited by 49
Statistical power of articles published in three health-psychology related journals.
Maddock, J. E., & Rossi, J. S. (2001) · Health Psychology · doi:10.1037//0278-6133.20.1.76 · Cited by 46
Effect Size Reporting Practices in Published Articles
Alhija, F. N.-A., & Levy, A. (2009) · Educational and Psychological Measurement · doi:10.1177/0013164408315266 · Cited by 43 · cited 3×
“Alhija and Levy (2009) reviewed the literature on re - porting effect sizes, and noted that the fifth edition of the Publication Manual of the American Psychological Association (APA, 2001) made stronger recommendations for reporting effect sizes than previous editions.”
Merenda, P. F. (2007) · Perceptual and Motor Skills · doi:10.2466/pms.104.1.3-20 · Cited by 26 · cited 7×
“the decline in the number of psychometricians and quantitative methodologists observed by Merenda (2007). A final recommendation, then, is to redouble current efforts to educate psychology students about the importance of power and sample size.”
InfluentialSample Size in Psychological Research
Holmes, C. B. (1979) · Perceptual and Motor Skills · doi:10.2466/pms.1979.49.1.283 · Cited by 14 · cited 16×
“Holmes (1979, 1983) used the following procedure for determining sample sizes reported in 1955 and 1977 journal articles, which was also used in the 1995 and 2006 data collections.”
InfluentialSample Size in Four Areas of Psychological Research
Holmes, C. B. (1983) · Transactions of the Kansas Academy of Science · doi:10.2307/3627914 · Cited by 12 · cited 11×
“Holmes (1979, 1983) used the following procedure for determining sample sizes reported in 1955 and 1977 journal articles, which was also used in the 1995 and 2006 data collections.”
InfluentialSample Size in Non-Apa Journals
Holmes, C. B., Holmes, J. R., & Fanning, J. J. (1981) · The Journal of Psychology · doi:10.1080/00223980.1981.9915273 · Cited by 4 · cited 6×
“To account for this, Holmes, et al .”
InfluentialPsychology and the Scientific Method
Horst, P. (1931) · The Journal of Philosophy · doi:10.2307/2017049 · Cited by 2
Research works printed but not linked: 4
- American Psychological Association, 2001, Publication manual
- Cochrane R., 1974, Bulletin of the British Psychological Society, 27, 117
- 10.1037/0278-6133.20.1.76 · doi:10.1037/0278-6133.20.1.76 · Paper page
- Raudenbush S. W., 2004, HLM 6: Hierarchical linear and nonlinear modeling
Cited by
Effect size guidelines for individual differences researchers
Gignac, G. E., & Szodorai, E. T. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2016.06.069 · Cited by 2,860
The association between adolescent well-being and digital technology use
Orben, A., & Przybylski, A. K. (2019) · Nature Human Behaviour · doi:10.1038/s41562-018-0506-1 · Cited by 1,500
de Winter, J. C. F., Gosling, S. D., & Potter, J. (2016) · Psychological Methods · doi:10.1037/met0000079 · Cited by 1,201
“Although sample sizes in psychological research vary widely, we used N = 200 because this is in line with typical sample sizes used in applied and personality psychology (Fraley & Vazire, 2014; Marszalek et al., 2011).”
BackgroundMethodThe rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Metadata
- Authors
- Marszalek, J. M., Barber, C., Kohlhart, J., & Cooper, B. H.
- Affiliations
- University of Missouri–Kansas City (US)
- Emporia State University (US)
- Corresponding author
- Marszalek, Jacob M.
- Venue
- Perceptual and Motor Skills
- Publisher
- SAGE Publications
- Published
- 2011-04
- Volume
- 112
- Issue
- 2
- Pages
- 331-348
- ISSN
- 0031-5125 (ISSN-L)
- 1558-688X
- OpenAlex
- W2126053474
- Cited by
- 167
- References
- 29
- Citations by year
- 2026: 6 · 2025: 9 · 2024: 10 · 2023: 11 · 2022: 9 · 2021: 10 · 2020: 17 · 2019: 21 · 2018: 11 · 2017: 15 · 2016: 19 · 2015: 12 · 2014: 8 · 2013: 6 · 2012: 3
- FWCI
- 7.03
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Perceptual and Motor Skills (publishedVersion)
- PubMed (publishedVersion)
- MOspace Institutional Repository (University of Missouri) (submittedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- sample size, statistical power, methodological critique, research credibility, APA journals, methodological recommendations
- MeSH terms
- Data Interpretation, Statistical, Forecasting, Humans, Periodicals as Topic, Psychology, Publishing, Research, Reproducibility of Results, Bias, Linear Models, Sample Size, Statistics, Nonparametric, Editorial Policies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.