Sample Size in Psychological Research over the Past 30 Years
The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
The American Psychological Association (APA) Task Force on Statistical Inference was formed in 1996 in response to a growing body of research demonstrating methodological issues that threatened the credibility of psychological research, and made recommendations to address them. One issue was the small, even dramatically inadequate, size of samples used in studies published by leading journals. The present study assessed the progress made since the Task Force's final report in 1999. Sample sizes reported in four leading APA journals in 1955, 1977, 1995, and 2006 were compared using nonparametric statistics, while data from the last two waves were fit to a hierarchical generalized linear growth model for more in-depth analysis. Overall, results indicate that the recommendations for increasing sample sizes have not been integrated in core psychological research, although results slightly vary by field. This and other implications are discussed in the context of current methodological critique and practice.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Third, our sample is 50 times larger than the average sample of studies published in high-impact psychology journals (30, 31).”
From Cognitive and noncognitive predictors of success (2019) · cites this paper as [31]
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Answering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225 · cited 1×
“Agreement was measured with Krippendorff’s alpha, α K , using the KALPHA SPSS (now PASW) macro (Hayes & Krippendorff, 2007), treating sample size as a ratio level measurement and performing 1,000 bootstrapping samples.”
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824 · cited 5×
“(1981) parallel the surveys of observed power reported in the literature (Cohen, 1962; Sedlmeier & Gigerenzer, 1989; Rossi, 1990; Maddock & Ros-si, 2001; Maxwell, 2004), and even overlap in specific journals (i.e., Journal of Abnormal Psychology ).”
InfluentialNonparametric statistics for non-statisticians: a step-by-step approach
(2010) · Choice Reviews Online · doi:10.5860/choice.47-3859 · Cited by 1,021
Nonparametric Statistics for Non‐Statisticians: A Step‐by‐Step Approach
Corder, G. W., & Foreman, D. I. (2009) · Wiley eBooks · doi:10.1002/9781118165881 · Cited by 954
Research works printed but not linked: 4
- American Psychological Association, 2001, Publication manual
- Cochrane R., 1974, Bulletin of the British Psychological Society, 27, 117
- 10.1037/0278-6133.20.1.76 · doi:10.1037/0278-6133.20.1.76 · Paper page
- Raudenbush S. W., 2004, HLM 6: Hierarchical linear and nonlinear modeling
Cited by
Effect size guidelines for individual differences researchers
Gignac, G. E., & Szodorai, E. T. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2016.06.069 · Cited by 2,860
The association between adolescent well-being and digital technology use
Orben, A., & Przybylski, A. K. (2019) · Nature Human Behaviour · doi:10.1038/s41562-018-0506-1 · Cited by 1,500
de Winter, J. C. F., Gosling, S. D., & Potter, J. (2016) · Psychological Methods · doi:10.1037/met0000079 · Cited by 1,201
“Although sample sizes in psychological research vary widely, we used N = 200 because this is in line with typical sample sizes used in applied and personality psychology (Fraley & Vazire, 2014; Marszalek et al., 2011).”
BackgroundMethodThe rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847196 · Cited by 588
Top 10 Replicated Findings From Behavioral Genetics
Plomin, R., DeFries, J. C., Knopik, V. S., & Neiderhiser, J. M. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691615617439 · Cited by 517
Anderson, S. F., Kelley, K., & Maxwell, S. E. (2017) · Psychological Science · doi:10.1177/0956797617723724 · Cited by 406
Continuously Cumulating Meta-Analysis and Replicability
Braver, S. L., Thoemmes, F. J., & Rosenthal, R. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614529796 · Cited by 297
Meaningful Mediation Analysis: Plausible Causal Inference and Informative Communication
Pieters, R. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx081 · Cited by 266
A tutorial on Bayes Factor Design Analysis using an informed prior
Stefan, A. M., Gronau, Q. F., Schönbrodt, F. D., & Wagenmakers, E.-J. (2019) · Behavior Research Methods · doi:10.3758/s13428-018-01189-8 · Cited by 284
Statistical Rituals: The Replication Delusion and How We Got There
Gigerenzer, G. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918771329 · Cited by 263
Linxen, S., Sturm, C., Brühlmann, F., Cassau, V., Opwis, K., & Reinecke, K. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445488 · Cited by 254
“in 2011 [58] for conventional laboratory studies) means that these studies are often underpowered and fail to replicate [4].”
BackgroundTackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., Oltmanns, T. F., & Shrout, P. E. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617690042 · Cited by 251
Brandt, M. J. (2013) · Journal of Personality and Social Psychology · doi:10.1037/a0031751 · Cited by 230
Quantitative Methods in Psychology
(1938) · Nature · doi:10.1038/141613a0 · Cited by 226
The Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles
O’Boyle, E. H., Banks, G. C., & Gonzalez-Mulé, E. (2017) · Journal of Management · doi:10.1177/0149206314527133 · Cited by 246
Expectations for Replications: Are Yours Realistic?
Stanley, D. J., & Spence, J. R. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528518 · Cited by 209
Finkel, E. J., Eastwick, P. W., & Reis, H. T. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000007 · Cited by 202
Cognitive and noncognitive predictors of success
Duckworth, A. L., Quirk, A., Gallop, R., Hoyle, R. H., Kelly, D. R., & Matthews, M. D. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1910510116 · Cited by 170
“Third, our sample is 50 times larger than the average sample of studies published in high-impact psychology journals (30, 31).”
An Introduction to Registered Replication Reports at Perspectives on Psychological Science
Simons, D. J., Holcombe, A. O., & Spellman, B. A. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614543974 · Cited by 184
Researchers’ Intuitions About Power in Psychological Research
Bakker, M., Hartgerink, C. H. J., Wicherts, J. M., & van der Maas, H. L. J. (2016) · Psychological Science · doi:10.1177/0956797616647519 · Cited by 146
Improving on Adjusted R-Squared
Karch, J. D. (2020) · Collabra Psychology · doi:10.1525/collabra.343 · Cited by 139
P-hacking and publication bias interact to distort meta-analytic effect size estimates
Friese, M., & Frankenbach, J. (2020) · Psychological Methods · doi:10.1037/met0000246 · Cited by 122
Smith, M. M., Pereira, S. M. P., Chan, L., Rose, C., & Shafran, R. (2018) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-018-0253-x · Cited by 120
“Evaluating Interventions Evaluations of interventions are limited and typically associated with methodological issues including small sample sizes (Marszalek et al. 2011), a lack of intervention integrity tracking (Kryzak et al.”
Backgroundvon Hippel, P. T. (2016) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1080/10705511.2015.1047931 · Cited by 72
Direction of effects in mediation analysis.
Wiedermann, W., & von Eye, A. (2015) · Psychological Methods · doi:10.1037/met0000027 · Cited by 77
Memory Detection 2.0: The First Web-Based Memory Detection Test
Kleinberg, B., & Verschuère, B. (2015) · PLoS ONE · doi:10.1371/journal.pone.0118715 · Cited by 82
“A matter of concern is that most effects in psychology are of moderate effect size, while most psychology studies do not have sufficiently large sample sizes to detect such effects [23].”
BackgroundThe risks of and barriers to social procurement in construction: a supply chain perspective
Loosemore, M., Alkilani, S. Z., & Mathenge, R. (2019) · Construction Management and Economics · doi:10.1080/01446193.2019.1687923 · Cited by 90
The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates
Nuijten, M. B., van Assen, M. A. L. M., Veldkamp, C. L. S., & Wicherts, J. M. (2015) · Review of General Psychology · doi:10.1037/gpr0000034 · Cited by 69
“…power is .40 –.47 across studies in behavioral ecology (Jennions 2 Estimated given a two independent samples comparison, assuming an effect size of d .50 (based on estimates from meta-analyses) and a total sample size of 40, the median total sample size in psychology (Marszalek et al., 2011).”
BackgroundA Concise Set of Core Recommendations to Improve the Dependability of Psychological Research
Lishner, D. A. (2015) · Review of General Psychology · doi:10.1037/gpr0000028 · Cited by 77
Elliot, A. J. (2018) · Review of General Psychology · doi:10.1037/gpr0000170 · Cited by 79
Adelman, L., Leidner, B., Ünal, H., Nahhas, E. A. H., & Shnabel, N. (2016) · Personality and Social Psychology Bulletin · doi:10.1177/0146167216662868 · Cited by 82
Evolutionary psychological consumer research: Bold, bright, but better with behavior
Otterbring, T., Sundie, J. M., Li, Y. J., & Hill, S. E. (2020) · Journal of Business Research · doi:10.1016/j.jbusres.2020.07.010 · Cited by 73
Cheng, C., Yang, C., Inder, K., & Chan, S. W. (2020) · Journal of Nursing Scholarship · doi:10.1111/jnu.12540 · Cited by 68
Benefits of Open and High-Powered Research Outweigh Costs
LeBel, E. P., Campbell, L., & Loving, T. J. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000049 · Cited by 72
A Bayesian approach to mitigation of publication bias
Guan, M., & Vandekerckhove, J. (2015) · Psychonomic Bulletin & Review · doi:10.3758/s13423-015-0868-6 · Cited by 68
Renkewitz, F., & Keiner, M. (2019) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000386 · Cited by 55
Vargha, A., Bergman, L. R., & Takács, S. (2016) · Journal for Person-Oriented Research · doi:10.17505/jpor.2016.08 · Cited by 62
Open science, closed doors: The perils and potential of open science for research in practice
Guzzo, R. A., Schneider, B., & Nalbantian, H. R. (2022) · Industrial and Organizational Psychology · doi:10.1017/iop.2022.61 · Cited by 61
“Marszalek et al. (2011) tracked four journals over time (Journal of Applied Psychology among them) and found median sample sizes of 48, 32, and 40, respectively, in 1977, 1995, and 2006.”
BackgroundBarber, S. J., Lopez, N., Cadambi, K., & Alferez, S. (2020) · Cognition · doi:10.1016/j.cognition.2020.104267 · Cited by 52
Heteroscedasticity as a Basis of Direction Dependence in Reversible Linear Regression Models
Wiedermann, W., Artner, R., & von Eye, A. (2017) · Multivariate Behavioral Research · doi:10.1080/00273171.2016.1275498 · Cited by 61
“Sample size was fixed at n = 200, which is considered common in the psychological sciences (Marszalek et al., 2011; see also Jaccard &Wan, 1995).”
BackgroundWorld Citizens on the Periphery: Threat and Identification with Global Society
Gorman, B., & Seguin, C. (2018) · American Journal of Sociology · doi:10.1086/699652 · Cited by 50
“20 In all cases that we analyze, each comparison group (before and after the attack) had at least 50 respondents, a group sample size generally larger than those used in psychology experiments (see Marszalek et al. 2011).”
Questionable research practices may have little effect on replicability
Ulrich, R., & Miller, J. O. (2020) · eLife · doi:10.7554/elife.58237 · Cited by 53
DeBolt, M. C., Rhemtulla, M., & Oakes, L. M. (2020) · Infancy · doi:10.1111/infa.12337 · Cited by 49
Too Good to be False: Nonsignificant Results Revisited
Hartgerink, C. H. J., Wicherts, J. M., & van Assen, M. A. L. M. (2017) · Collabra: Psychology · doi:10.1525/collabra.71 · Cited by 45
When and Why to Replicate: As Easy as 1, 2, 3?
Field, S. M., Hoekstra, R., Bringmann, L. F., & van Ravenzwaaij, D. (2019) · Collabra Psychology · doi:10.1525/collabra.218 · Cited by 56
Daytime dynamic of spontaneous expressions of pedestrians in an urban forest park
Wei, H., Hauer, R. J., & Guo, S. (2021) · Urban forestry & urban greening · doi:10.1016/j.ufug.2021.127326 · Cited by 47
Anderson, S. F. (2020) · Psychological Methods · doi:10.1037/met0000248 · Cited by 46
“60 per group, reflecting the lower and higher ends of typical sample sizes for designs of this type in psychology (e.g., Marszalek, Barber, Kohlhart, & Holmes, 2011).”
Graphical Descriptives: A Way to Improve Data Transparency and Methodological Rigor in Psychology
Tay, L., Parrigon, S., Huang, Q., & LeBreton, J. M. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616663875 · Cited by 48
“This may be especially important when smaller sample sizes are used (previous reviews suggest median N s < 200 in psychological research; Marszalek, Barber, Kohlhart, & Holmes, 2011; Shen et al., 2011) and group means—which are typically used—can be vulnerable to outliers.”
BackgroundEffect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis
Nuijten, M. B., van Assen, M. A. L. M., Augusteijn, H. E. M., Crompvoets, E. A. V., & Wicherts, J. M. (2020) · Journal of Intelligence · doi:10.3390/jintelligence8040036 · Cited by 45
“Despite repeated recommendations to change this, there seems to have been no overall improvement (Fraley and Vazire 2014; Hartgerink et al. 2017; Marszalek et al. 2011; Maxwell 2004; Stanley et al. 2018; Szucs and Ioannidis 2017; but see Maddock and Rossi 2001; Rossi 1990; van Aert et al. 2019).”
BackgroundResultLeidner, B. (2015) · Personality and Social Psychology Bulletin · doi:10.1177/0146167215607195 · Cited by 42
George, M., & Loosemore, M. (2018) · Construction Management and Economics · doi:10.1080/01446193.2018.1535713 · Cited by 44
“The response rate of 23% is strong for studies in applied psychology which address potentially sensitive areas (Marszalek et al. 2011), especially given the highly dispersed and transient nature of our respondents’ work sites and the fact that most do not use internet at work.”
BackgroundOkada, K. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0760-y · Cited by 41
Bakker, M., Veldkamp, C. L. S., van den Akker, O. R., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., & Wicherts, J. M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0236079 · Cited by 45
“Although we found that more power analyses are reported and that the planned sample sizes were larger than earlier found in psychological studies (cell sizes of 20–24; [2, 44, 45]), most of the incorporated studies are still unlikely to detect a true but smaller than medium effect.”
ALPHA VALUES AS A FUNCTION OF SAMPLE SIZE, EFFECT SIZE, AND POWER: ACCURACY OVER INFERENCE1
Bradley, M. T., & Brand, A. (2013) · Psychological Reports · doi:10.2466/03.49.pr0.112.3.835-844 · Cited by 44
Therapeutic alliance over the course of child trauma therapy from three different perspectives
Zorzella, K. P. M., Rependa, S. L., & Muller, R. T. (2017) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2017.02.032 · Cited by 41
Are they for us or against us? How intergroup metaperceptions shape foreign policy attitudes
O’Brien, T. C., Leidner, B., & Tropp, L. R. (2017) · Group Processes & Intergroup Relations · doi:10.1177/1368430216684645 · Cited by 40
Equivalence testing for linear regression.
Campbell, H. (2023) · Psychological Methods · doi:10.1037/met0000596 · Cited by 37
Ostinato rigore : establishing methodological rigour in quantitative research
Laher, S. (2016) · South African Journal of Psychology · doi:10.1177/0081246316649121 · Cited by 48
“Where possible, extraneous variables arising from sampling should be anticipated and controlled for or at the very least examined in addition to examining the research questions particular to the study (Marszalek et al., 2011).”
BackgroundMeaningful change definitions: sample size planning for experimental intervention research
Gruijters, S. L. K., & Peters, G.-J. (2020) · Psychology and Health · doi:10.1080/08870446.2020.1841762 · Cited by 35
“…if only one study has been conducted, given the small sample sizes and low statistical power that have long been customary in the psychological literature (Marszalek et al., 2011), the effect size estimate obtained in that study likely originates from an exceedingly wide sampling distribution.”
BackgroundProactive Coping, Time Perspective and Life Satisfaction: A Study on Emerging Adulthood
Dwivedi, A., & Rastogi, R. (2017) · Journal of Health Management · doi:10.1177/0972063417699689 · Cited by 38
Campbell, H., & Lakens, D. (2020) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12201 · Cited by 33
“The R-code for analysis of the McCambridge et al. (2019) data is:”
MethodResultSchuster, R., Kaiser, T., Terhorst, Y., Meßner, E.-M., Strohmeier, L.-M., & Laireiter, A. (2021) · Psychological Medicine · doi:10.1017/s003329172100129x · Cited by 31
A comparison of methods for the analysis of binomial clustered outcomes in behavioral research
Ferrari, A., & Comelli, M. (2016) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2016.10.005 · Cited by 31
Improving on Adjusted R-SquaredPreprint
Karch, J. D. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/v8dz5 · Cited by 29
Rim, H., Swenson, R. D., & Anderson, B. (2019) · Journal of Applied Communication Research · doi:10.1080/00909882.2019.1654125 · Cited by 29
Brand, A., & Bradley, M. T. (2016) · Psychological Reports · doi:10.1177/0033294115625265 · Cited by 25
“Hence these results reflect the findings from the survey of sample sizes conducted by Marszalek et al. (2011).”
ResultBeyond Pearson’s Correlation: Modern Nonparametric Independence Tests for Psychological Research
Karch, J. D., Alonso, A. F. P., & Bergsma, W. (2024) · Multivariate Behavioral Research · doi:10.1080/00273171.2024.2347960 · Cited by 30
“Marszalek et al. (2011) found that the most common sample sizes in psychology range from 10 to 140, while K € uhberger et al. (2014) found a range from 1 to 100.”
Alessandroni, N., & Rodríguez, C. (2020) · Psicologia Reflexão e Crítica · doi:10.1186/s41155-020-00154-9 · Cited by 24
“…is a severe problem, as it undermines the statistical power of experimental explanatory studies by making the results neither meaningful nor reproducible, especially in the case of psychological and neuroscientific literature (e.g., Bakker et al., 2012; Button et al., 2013; Marszalek et al., 2011).”
BackgroundCrowdsourcing in Cognitive and Systems Neuroscience
Johnson, B. P., Dayan, E., Censor, N., & Cohen, L. G. (2021) · The Neuroscientist · doi:10.1177/10738584211017018 · Cited by 24
Twenty-Year Trends of Authorship and Sampling in Applied Biomechanics Research
Knudson, D. V. (2012) · Perceptual and Motor Skills · doi:10.2466/11.pms.114.1.16-20 · Cited by 23
Log-linear models to evaluate direction of effect in binary variables
Wiedermann, W., & von Eye, A. (2017) · Statistical Papers · doi:10.1007/s00362-017-0936-2 · Cited by 22
Schuster, R., Schreyer, M. L., Kaiser, T., Berger, T., Klein, J. P., Moritz, S., Laireiter, A.-R., & Trutschnig, W. (2020) · Internet Interventions · doi:10.1016/j.invent.2020.100313 · Cited by 21
“Even though this phenomenon does not present uniformly (Maddock and Rossi, 2001; Marszalek et al., 2011), the practice of underpowered studies in the clinical field can be described as widespread and hard to change – ultimately increasing the risk to aggregate false findings in meta analytic…”
BackgroundPreliminary Evidence for How the Behavioral Immune System Predicts Juror Decision-Making
Brown, M., Rodriguez, D. N., Gretak, A. P., & Berry, M. A. (2017) · Evolutionary Psychological Science · doi:10.1007/s40806-017-0102-z · Cited by 22
“One possibility for these null results could be related to insufficient power; although our sample size fares well in comparison to samples typically relied on in the recent past (see, e.g., Marszalek et al. 2011), it still offers limited ability to detect small effects.”
ResultDirectionality of Effects in Causal Mediation Analysis
Wiedermann, W., & von Eye, A. (2016) · Wiley series in probability and statistics · doi:10.1002/9781118947074.ch4 · Cited by 21
Australian Flight Crews’ Trust in Voluntary Reporting Systems and Just Culture Policies
McMurtrie, K. J., & Molesworth, B. R. C. (2018) · Aviation Psychology and Applied Human Factors · doi:10.1027/2192-0923/a000131 · Cited by 22
Albaum, C. S., Sellitto, T., Vashi, N., Bohr, Y. M., & Weiss, J. A. (2023) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-023-06083-7 · Cited by 19
“Relatedly, small sample size is a ubiquitous issue in psychological research because of the connection to sta - tistical power and the capacity to detect true small-sized effects (Marszalek et al., 2011 ).”
BackgroundRaoust, G., Kajonius, P., & Hansson, S. R. (2023) · Scientific Reports · doi:10.1038/s41598-023-32658-6 · Cited by 17
List, A. (2023) · The Journal of Experimental Education · doi:10.1080/00220973.2023.2174487 · Cited by 17
“It is notable that the “ hundreds ” of participants described in all of the quantitative excerpts likely constituted a sufficient sample size for the analyses suggested (e.g., correlation, group comparisons) and were typical of samples used in psychology studies (Marszalek et al., 2011).”
Resultp-Hacking and Publication Bias Interact to Distort Meta-Analytic Effect Size EstimatesPreprint
Friese, M., & Frankenbach, J. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/bvm58 · Cited by 15
Renkewitz, F., & Keiner, M. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/w94ep · Cited by 19
ROCZNIKI PSYCHOLOGICZNE / ANNALS OF PSYCHOLOGY
Soroko, E., & Psychologii, I. (2013) · Cited by 14
Twenty Years of Authorship, Sampling, and References in Kinesiology Research Reports
Knudson, D. V. (2017) · International Journal of Kinesiology in Higher Education · doi:10.1080/24711616.2017.1282760 · Cited by 14
Tinkler, J., Clay‐Warner, J., & Alinor, M. (2018) · Journal of Interpersonal Violence · doi:10.1177/0886260518798356 · Cited by 14
Dynamic modelling of travellers’ social interactions and social learning
Sunitiyoso, Y., Avineri, E., & Chatterjee, K. (2013) · Journal of Transport Geography · doi:10.1016/j.jtrangeo.2013.05.012 · Cited by 13
Robust statistical methods and the credibility movement of psychological science
Sladekova, M., & Field, A. P. (2025) · PeerJ · doi:10.7717/peerj.20043 · Cited by 12
“…of 55 across four areas of psychology (Abnormal Psychology, Applied Psychology, Developmental Psychology, and Experimental Psychology), while Marszalek et al. (2011) found the median sample size across these areas to be even lower ( n = 40) when replicating Holmes’s (1983) study three…”
Bateman, J. E., & Birney, D. P. (2019) · Acta Psychologica · doi:10.1016/j.actpsy.2019.102893 · Cited by 12
Beyond statistical significance: Five principles for the new era of data analysis and reporting
Wedel, M., & Gal, D. (2023) · Journal of Consumer Psychology · doi:10.1002/jcpy.1379 · Cited by 13
“Sample size matters: the lack of power resulting from small samples (Cohen, 1962; Marszalek et al., 2012; Maxwell, 2004) negatively affects the reliability of study results (Greenland, 2012).”
Reliability and correlates of intra-individual variability in the oculomotor system
Perquin, M. N., & Bompas, A. (2019) · Journal of Eye Movement Research · doi:10.16910/jemr.12.6.11 · Cited by 11
Revisión sistemática del compromiso escolar en educación secundaria
Sáez-Delgado, F., Parra, V. S., Pérez-Salaz, C., Ramírez, P., & Zañartu, I. (2023) · Interdisciplinaria Revista de Psicología y Ciencias Afines · doi:10.16888/interd.2023.40.2.3 · Cited by 11
Bakker, M. (2015) · Cortex · doi:10.1016/j.cortex.2015.07.006 · Cited by 11
Social Vulnerability in Williams Syndrome
Riby, D. M., Ridley, E., Lough, E., & Hanley, M. (2017) · International review of research in developmental disabilities · doi:10.1016/bs.irrdd.2017.05.001 · Cited by 13
Integrative Data Analysis for Research in Developmental Psychopathology
Mun, E., Jiao, Y., & Xie, M. (2016) · Wiley eBooks · doi:10.1002/9781119125556.devpsy123 · Cited by 11
Canadian psychology department participant pools: Closing for the season?
Sharpe, D., & Poets, S. (2017) · Canadian Psychology/Psychologie canadienne · doi:10.1037/cap0000090 · Cited by 12
Cross-Cultural Variability in Self-Continuity Warranting Strategies
Dunlop, W. L., & Walker, L. J. (2014) · Journal of Language and Social Psychology · doi:10.1177/0261927x14555873 · Cited by 10
Investigating bias in squared regression structure coefficients
Nimon, K., Zientek, L. R., & Thompson, B. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00949 · Cited by 10
“The sample sizes were chosen that were reflective of articles published in education, psychology, and counseling (Kieffer et al., 2001; Marszalek et al., 2011).”
BackgroundThird moment-based causal inference
Wiedermann, W. (2022) · Behaviormetrika · doi:10.1007/s41237-021-00154-8 · Cited by 9
Too Small to Succeed: Small Samples and the p-Value Problem
Aguirre-Urreta, M. I., Rönkkö, M., & McIntosh, C. N. (2024) · ACM SIGMIS Database the DATABASE for Advances in Information Systems · doi:10.1145/3685235.3685238 · Cited by 9
“First, estimates from small samples are highly imprecise (Brand & Bradley, 2016; Marszalek et al., 2011) and thus make the results of an individual study less trustworthy—and less publishable.”
Metadata
- Authors
- Marszalek, J. M., Barber, C., Kohlhart, J., & Cooper, B. H.
- Affiliations
- University of Missouri–Kansas City (US)
- Emporia State University (US)
- Corresponding author
- Marszalek, Jacob M.
- Venue
- Perceptual and Motor Skills
- Publisher
- SAGE Publications
- Published
- 2011-04
- Volume
- 112
- Issue
- 2
- Pages
- 331-348
- ISSN
- 0031-5125 (ISSN-L)
- 1558-688X
- OpenAlex
- W2126053474
- Cited by
- 167
- References
- 29
- Citations by year
- 2026: 6 · 2025: 9 · 2024: 10 · 2023: 11 · 2022: 9 · 2021: 10 · 2020: 17 · 2019: 21 · 2018: 11 · 2017: 15 · 2016: 19 · 2015: 12 · 2014: 8 · 2013: 6 · 2012: 3
- FWCI
- 7.03
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Perceptual and Motor Skills (publishedVersion)
- PubMed (publishedVersion)
- MOspace Institutional Repository (University of Missouri) (submittedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- sample size, statistical power, methodological critique, research credibility, APA journals, methodological recommendations
- MeSH terms
- Data Interpretation, Statistical, Forecasting, Humans, Periodicals as Topic, Psychology, Publishing, Research, Reproducibility of Results, Bias, Linear Models, Sample Size, Statistics, Nonparametric, Editorial Policies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.