Facts Are More Important Than Novelty: Replication in the Education Sciences
Abstract
Despite increased attention to methodological rigor in education research, the field has focused heavily on experimental design and not on the merit of replicating important results. The present study analyzed the complete publication history of the current top 100 education journals ranked by 5-year impact factor and found that only 0.13% of education articles were replications. Contrary to previous findings in medicine, but similar to psychology, the majority of education replications successfully replicated the original studies. However, replications were significantly less likely to be successful when there was no overlap in authorship between the original and replicating articles. The results emphasize the importance of third-party, direct replications in helping education research improve its ability to shape education policy and practice.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Novel or innovative results are more likely to be published than replications or incremental extensions of existing findings.
“Novel or innovative results are more likely to be published than replications or incremental extensions of existing findings 13-15 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [15]
Cites
2 retracted
Wakefield, A., Murch, S., Anthony, A., Linnell, J., Casson, D., Malik, M., Berelowitz, M., Dhillon, A., Thomson, M., Harvey, P., et al. (1998) · The Lancet · doi:10.1016/s0140-6736(97)11096-0 · Cited by 3,006 · see citing works
RetractedLancet, T. (2010) · The Lancet · doi:10.1016/s0140-6736(10)60175-4 · Cited by 472 · see citing works
Retracted(1963) · Medical Entomology and Zoology · Cited by 15,085 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · see citing works
Journal of social behavior and personality
(1986) · Personality and Individual Differences · doi:10.1016/0191-8869(86)90025-5 · Cited by 6,751 · see citing works
An Introduction to Categorical Data Analysis
Agresti, A. (2006) · Wiley series in probability and statistics · doi:10.1002/0470114754 · Cited by 6,697 · see citing works
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Moffitt, R. (2009) · American Economic Review · doi:10.1257/aer.99.2.660 · Cited by 4,888 · see citing works
Langenberg, D. N. (2000) · Cited by 4,325 · see citing works
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003 · see citing works
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200 · see citing works
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 3× · see citing works
“The concern over replication exists in many research domains, including advertising (Madden et al., 1995), biology (Begley & Ellis, 2012; Powell, 2007), economics (Anderson, Greene, McCullough, & Vinod, 2005; Dewald, Thursby, & Anderson, 1986; Kane, 1984), library sciences (Winters, 1996),…”
ResultInfluentialMisinformation and Its Correction: Continued Influence and Successful Debiasing
Lewandowsky, S., Ecker, U. K. H., Seifert, C. M., Schwarz, N., & Cook, J. (2012) · Psychological Science in the Public Interest · doi:10.1177/1529100612451018 · Cited by 2,920 · see citing works
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · see citing works
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989 · see citing works
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · see citing works
Plucker, J. A., Beghetto, R. A., & Dow, G. T. (2004) · Educational Psychologist · doi:10.1207/s15326985ep3902_1 · Cited by 1,794 · see citing works
Self-citationExperimenter Effects in Behavioral Research
McGuigan, F. J., & Rosenthal, R. (1968) · The American Journal of Psychology · doi:10.2307/1420816 · Cited by 1,639 · see citing works
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465 · see citing works
Reproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · see citing works
Statistical significance in psychological research.
Lykken, D. T. (1968) · Psychological bulletin · doi:10.1037/h0026141 · Cited by 1,363 · see citing works
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321 · see citing works
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301 · see citing works
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248 · see citing works
Experimenter Effects in Behavioral Research.
Titchener, J. L. (1967) · Archives of Internal Medicine · doi:10.1001/archinte.1967.00300050109027 · Cited by 1,207 · see citing works
Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
Schmidt, S. (2009) · Review of General Psychology · doi:10.1037/a0015108 · Cited by 1,125 · see citing works
Peer-review practices of psychological journals: The fate of published articles, submitted again
Peters, D. P., & Ceci, S. J. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011183 · Cited by 1,079 · cited 1× · see citing works
“…& Canada, 2013; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995), low reliability among peer reviewers (e.g., Cicchetti, 1991; Cole, 1992; D. Peters & Ceci, 1982), hypothesizing after the results are known (e.g., N. Kerr, 1998; Makel, 2014; Maxwell, 2004), misuse of statistical tests and…”
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042 · see citing works
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0022790 · Cited by 977 · see citing works
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 4× · see citing works
“…(e.g., Bakker, van Dijk, & Wicherts, 2012; Bozarth & Roberts, 1972; Fanelli, 2010, 2012; Pigott, Valentine, Polanin, Williams, & Canada, 2013; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995), low reliability among peer reviewers (e.g., Cicchetti, 1991; Cole, 1992; D. Peters & Ceci,…”
BackgroundInfluentialThe test of significance in psychological research.
Bakan, D. (1966) · Psychological Bulletin · doi:10.1037/h0020412 · Cited by 957 · see citing works
Bem, D. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021524 · Cited by 893 · see citing works
The demon-haunted world: science as a candle in the dark
(1996) · Choice Reviews Online · doi:10.5860/choice.34-0277 · Cited by 825 · see citing works
Revised standards for statistical evidence
Johnson, V. E. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1313476110 · Cited by 808 · see citing works
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800 · see citing works
Autism and measles, mumps, and rubella vaccine: no epidemiological evidence for a causal association
Taylor, B., Miller, E., Farrington, C., Petropoulos, M.-C., Favot-Mayaud, I., Li, J., & Waight, P. A. (1999) · The Lancet · doi:10.1016/s0140-6736(99)01239-8 · Cited by 753 · see citing works
Replications in Psychology Research: How Often Do They Really Occur?
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460688 · Cited by 750 · see citing works
Self-citationThe fallacy of the null-hypothesis significance test.
Rozeboom, W. W. (1960) · Psychological Bulletin · doi:10.1037/h0042040 · Cited by 738 · see citing works
“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734 · see citing works
The Replication Recipe: What makes for a convincing replication?
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner‐Sorolla, R., Grange, J. A., Perugini, M., Spies, J. R., & Veer, A. V. '. (2013) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.10.005 · Cited by 703 · see citing works
Is the Replicability Crisis Overblown? Three Arguments Examined
Pashler, H., & Harris, C. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463401 · Cited by 697 · see citing works
Maxwell, S. E. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.2.147 · Cited by 655 · see citing works
An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science
Open Science Collaboration (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462588 · Cited by 605 · see citing works
Why Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563 · see citing works
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.2307/2282137 · Cited by 524 · see citing works
Cicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065675 · Cited by 496 · see citing works
Replication in Empirical Economics: The Journal of Money, Credit and Banking Project
Dewald, W. G., Thursby, J. G., & Anderson, R. G. (2016) · American Economic Review · Cited by 491 · see citing works
The Design of Replicated Studies
Lindsay, R. M., & Ehrenberg, A. S. C. (1993) · The American Statistician · doi:10.1080/00031305.1993.10475983 · Cited by 491 · see citing works
An Introduction to Categorical Data Analysis
Bilder, C. R., & Tebbs, J. M. (2008) · Journal of the American Statistical Association · doi:10.1198/jasa.2008.s251 · Cited by 458 · see citing works
The ironic effect of significant results on the credibility of multiple-study articles.
Schimmack, U. (2012) · Psychological Methods · doi:10.1037/a0029487 · Cited by 455 · see citing works
King, G. (1995) · PS Political Science & Politics · doi:10.1017/s1049096500057607 · Cited by 455 · see citing works
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453 · see citing works
Replications and Extensions in Marketing – Rarely Published But Quite Contrary
Hubbard, R., & Armstrong, J. (1994) · International Journal of Research in Marketing · doi:10.1016/0167-8116(94)90003-5 · Cited by 444 · see citing works
Financial Incentives and Student Achievement: Evidence from Randomized Trials *
Fryer, R. G. (2011) · The Quarterly Journal of Economics · doi:10.1093/qje/qjr045 · Cited by 416 · see citing works
Yong, E. (2012) · Nature · doi:10.1038/485298a · Cited by 406 · see citing works
Making science: between nature and society
(1993) · Choice Reviews Online · doi:10.5860/choice.30-3240 · Cited by 359 · see citing works
The (mis)reporting of statistical results in psychology journals
Bakker, M., & Wicherts, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0089-5 · Cited by 356 · see citing works
Unpublished results hide the decline effect
Schooler, J. W. (2011) · Nature · doi:10.1038/470437a · Cited by 335 · see citing works
Science publishing: The trouble with retractions
Van Noorden, R. (2011) · Nature · doi:10.1038/478026a · Cited by 328 · see citing works
Analyzing data: Sanctification or detective work?
Tukey, J. W. (1969) · American Psychologist · doi:10.1037/h0027108 · Cited by 321 · see citing works
Priming, Replication, and the Hardest Science
Cesario, J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613513470 · Cited by 320 · see citing works
Eden, D. (2002) · Academy of Management Journal · doi:10.5465/amj.2002.7718946 · Cited by 318 · see citing works
Replication research's disturbing trend
Evanschitzky, H., Baumgarth, C., Hubbard, R., & Armstrong, J. S. (2007) · Journal of Business Research · doi:10.1016/j.jbusres.2006.12.003 · Cited by 271 · see citing works
Eich, E. (2013) · Psychological Science · doi:10.1177/0956797613512465 · Cited by 258 · see citing works
Jasny, B. R., Chin, G. J., Chong, L. D., & VIGNIERI, S. N. (2011) · Science · doi:10.1126/science.334.6060.1225 · Cited by 248 · see citing works
Improving Validation Practices in “Omics” Research
Ioannidis, J. P. A., & Khoury, M. J. (2011) · Science · doi:10.1126/science.1211811 · Cited by 246 · see citing works
Manuscript Characteristics Which Influence Acceptance for Management and Social Science Journals.
Kerr, S. J., Tolliver, J. T., & Petree, D. (1977) · Academy of Management Journal · doi:10.2307/255467 · Cited by 239 · see citing works
Teacher Incentives and Student Achievement: Evidence from New York City Public Schools
Fryer, R. G. (2013) · Journal of Labor Economics · doi:10.1086/667757 · Cited by 236 · see citing works
Teacher Incentives and Student Achievement: Evidence from New York City Public Schools
Fryer, R. G. (2011) · National Bureau of Economic Research · doi:10.3386/w16850 · Cited by 233 · see citing works
Hubbard, R., & Vetter, D. E. (1996) · Journal of Business Research · doi:10.1016/0148-2963(95)00084-4 · Cited by 230 · see citing works
Frank, M. C., & Saxe, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460686 · Cited by 175 · see citing works
Sterling, T. D., Rosenbaum, W. L., & Jj, W. (1995) · The American Statistician · doi:10.2307/2684823 · Cited by 168 · see citing works
Replication in prevention science
Valentine, J. C., Biglan, A., Boruch, R. F., Castro, F. G., Collins, L. M., Flay, B. R., Kellam, S., Mościcki, E. K., & Schinke, S. P. (2011) · Prevention Science · doi:10.1007/s11121-011-0217-6 · Cited by 166 · see citing works
Signifying significant significance.
Bozarth, J. D., & Roberts, R. R. (1972) · American Psychologist · doi:10.1037/h0038034 · Cited by 164 · see citing works
LeBel, E. P., & Peters, K. R. (2011) · Review of General Psychology · doi:10.1037/a0025172 · Cited by 152 · see citing works
Lack of Association between Measles Virus Vaccine and Autism with Enteropathy: A Case-Control Study
Hornig, M., Briese, T., Buie, T. M., Bauman, M. L., Lauwers, G. Y., Siemetzki, U., Hummel, K., Rota, P. A., Bellini, W. J., O’Leary, J. J., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003140 · Cited by 150 · see citing works
King, G. (1995) · PS Political Science & Politics · doi:10.2307/420301 · Cited by 145 · see citing works
Editorial bias against replication research.
Neuliep, J. W., & Crandall, R. (1990) · Journal of social behavior and personality · Cited by 141 · see citing works
How Journal Editors View Replication Research
Madden, C. S., Easley, R. W., & Dunn, M. G. (1995) · Journal of Advertising · doi:10.1080/00913367.1995.10673490 · Cited by 130 · see citing works
Outcome-Reporting Bias in Education Research
Pigott, T. D., Valentine, J. C., Polanin, J. R., Williams, R. T., & Canada, D. D. (2013) · Educational Researcher · doi:10.3102/0013189x13507104 · Cited by 127 · see citing works
Reviewer bias against replication research.
Neuliep, J. W., & Crandall, R. (1993) · Journal of social behavior and personality · Cited by 116 · see citing works
Concepts of Relative Importance in Recent Scientific Literature
Kruskal, W., & Majors, R. (1989) · The American Statistician · doi:10.1080/00031305.1989.10475596 · Cited by 115 · see citing works
Covert communication in the psychological experiment
Rosenthal, R. (1968) · PsycEXTRA Dataset · doi:10.1037/e535742008-002 · Cited by 114 · see citing works
Covert communication in the psychological experiment.
Rosenthal, R. (1967) · Psychological Bulletin · doi:10.1037/h0024529 · Cited by 113 · see citing works
Furchtgott, E. (1984) · American Psychologist · doi:10.1037/0003-066x.39.11.1315.b · Cited by 106 · see citing works
Tracking Replicability as a Method of Post-Publication Open Evaluation
Hartshorne, J. K., & Schachner, A. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00008 · Cited by 102 · see citing works
Carpenter, S. (2012) · Science · doi:10.1126/science.335.6076.1558 · Cited by 101 · see citing works
Psychology’s Woes and a Partial Cure: The Value of Replication
Roediger, H. L., III (2012) · APS observer · Cited by 101 · see citing works
Reproducibility: The risks of the replication drive
Bissell, M. (2013) · Nature · doi:10.1038/503333a · Cited by 99 · see citing works
(1956) · New England Journal of Medicine · doi:10.1056/nejm195611012551809 · Cited by 96 · see citing works
Methodological Challenges in the Study of Primate Cognition
Tomasello, M., & Call, J. (2011) · Science · doi:10.1126/science.1213443 · Cited by 93 · see citing works
Incentive Pay Programs Do Not Affect Teacher Motivation or Reported Practices
Yuan, K., Le, V., McCaffrey, D. F., Marsh, J. A., Hamilton, L. S., Stecher, B. M., & Springer, M. G. (2012) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373712462625 · Cited by 92 · see citing works
Crocker, J., & Cooper, M. L. (2011) · Science · doi:10.1126/science.1216775 · Cited by 86 · see citing works
The Institute of Education Sciences: New Wine, New Bottles.
Whitehurst, G. J. (2003) · Cited by 68 · see citing works
Making Science: Between Nature and Society.
Fuchs, S., & Cole, S. R. (1994) · Social Forces · doi:10.2307/2579806 · Cited by 68 · see citing works
Why Selective Publication of Statistically Significant Results Can Be Effective
de Winter, J., & Happee, R. (2013) · PLoS ONE · doi:10.1371/journal.pone.0066463 · Cited by 67 · see citing works
Harnessing the Undiscovered Resource of Student Research Projects
Grahe, J. E., Reifman, A. S., Hermann, A. D., Walker, M., Oleson, K. C., Nario‐Redmond, M. R., & Wiebe, R. P. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459057 · Cited by 64 · see citing works
REPLICATION IN EXPERIMENTAL COMMUNICATION RESEARCH: AN ANALYSIS
Kelly, C. W., Chase, L. J., & Tucker, R. K. (1979) · Human Communication Research · doi:10.1111/j.1468-2958.1979.tb00646.x · Cited by 59 · see citing works
Introduction to the Special Section: Data, Data, Everywhere . . . Especially in My File Drawer
Spellman, B. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691611432124 · Cited by 58 · see citing works
An Investigation of the Prevalence of Replication Research in Human Factors
Jones, K. S., Derby, P. L., & Schmidlin, E. A. (2010) · Hum. Factors · doi:10.1177/0018720810384394 · Cited by 44 · see citing works
Replication as a Verification Technique in Survey Research: A Paradigm
Sorte, M. A. L. (1972) · Sociological Quarterly · doi:10.1111/j.1533-8525.1972.tb00805.x · Cited by 43 · see citing works
Replication in Field Biology: The Case of the Frog-Eating Bat
Ryan, M. J. (2011) · Science · doi:10.1126/science.1214532 · Cited by 43 · see citing works
Publication practices and the file drawer problem: A survey of published authors.
Rotton, J., Foos, P. W., Meek, L. V., & Levitt, M. J. (1995) · Journal of social behavior and personality · Cited by 41 · see citing works
Concepts of Relative Importance in Recent Scientific Literature
Kruskal, W., & Majors, R. (1989) · The American Statistician · doi:10.2307/2685157 · Cited by 36 · see citing works
Creativity is more than novelty: Reconsidering replication as a creativity act.
Makel, M. C., & Plucker, J. (2014) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/a0035811 · Cited by 35 · see citing works
Self-citationReplication, replication, replication
Ritchie, S. J., Wiseman, R. J., & French, C. C. (2012) · University of Hertfordshire Research Archive (University of Hertfordshire) · Cited by 33 · see citing works
The empirical march: Making science better at self-correction.
Makel, M. C. (2014) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/a0035803 · Cited by 32 · see citing works
Self-citationDisproportionate referrals for child academic/behavior problems: Replication and extension.
Tarnowski, K. J., Anderson, D. F., Drabman, R. S., & Kelly, P. A. (1990) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.58.2.240 · Cited by 29 · see citing works
Replicability as a publication criterion.
Lubin, A. (1957) · American Psychologist · doi:10.1037/h0039746 · Cited by 24 · see citing works
Powell, K. (2007) · PLoS Biology · doi:10.1371/journal.pbio.0050338 · Cited by 18 · see citing works
Replication and Non-Quantitative Research
Golden, M. A. (1995) · PS Political Science & Politics · doi:10.2307/420313 · Cited by 18 · see citing works
The Role of Data & Program Code Archives in the Future of Economic Research
Anderson, R. G., Vinod, H. D., Greene, W. H., & McCullough, B. D. (2005) · doi:10.20955/wp.2005.014 · Cited by 17 · see citing works
On not so replicated experiments and not so null results.
Rosenthal, R. (1969) · Journal of Consulting and Clinical Psychology · doi:10.1037/h0027231 · Cited by 17 · see citing works
Social desirability in children: An extension and replication.
Klein, E. B., Gould, L. J., & Corey, M. (1969) · Journal of Consulting and Clinical Psychology · doi:10.1037/h0027339 · Cited by 15 · see citing works
Hansen, W. B. (2011) · Prevention Science · doi:10.1007/s11121-011-0218-5 · Cited by 12 · see citing works
Welcoming Quality in Non-Significance and Replication Work, but Moving Beyond the p -Value
McBee, M. T., & Matthews, M. S. (2014) · Journal of Advanced Academics · doi:10.1177/1932202x14532177 · Cited by 12 · see citing works
The Role of Data and Program Code Archives in the Future of Economic Research
Anderson, R. G., Greene, W. H., & McCullough, B. D. (2005) · Cited by 12 · see citing works
Reflections on replications and the experimenter bias effect.
Levy, L. H. (1969) · Journal of Consulting and Clinical Psychology · doi:10.1037/h0027230 · Cited by 10 · see citing works
A modest proposal for encouraging replication.
Ahlgren, A. (1969) · American Psychologist · doi:10.1037/h0037798 · Cited by 10 · see citing works
Loevinger, J. (1968) · American Psychologist · doi:10.1037/h0020800 · Cited by 9 · see citing works
Change starts with journal editors: In response to Makel (2014).
McBee, M. T., & Matthews, M. S. (2014) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/a0035801 · Cited by 8 · see citing works
Pizzello, S. (2023) · University Press of Mississippi eBooks · doi:10.2307/jj.7193915.8 · Cited by 8 · see citing works
Repeating fundamental experiments.
Schlosberg, H. (1951) · American Psychologist · doi:10.1037/h0056148 · Cited by 7 · see citing works
Ostrom, T. H. (1971) · American Psychologist · doi:10.1037/h0020338 · Cited by 4 · see citing works
Contradictions in Highly Cited Clinical Research—Reply
(2005) · JAMA · doi:10.1001/jama.294.21.2696-a · Cited by 3 · see citing works
An Example of Replication of an Experiment for Increased Reliability
Peters, C. C. (1938) · The Journal of Educational Research · doi:10.1080/00220671.1938.10880802 · Cited by 2 · see citing works
Publishing Replications: OK, Let's Try It
Meier, K. J. (1995) · PS Political Science & Politics · doi:10.2307/420515 · Cited by 2 · see citing works
But i don't have anything to write about
Winters, B. A. (1996) · Library Acquisitions Practice & Theory · doi:10.1016/s0364-6408(96)00068-3 · Cited by 2 · see citing works
Commentaries on Replication in Prevention Science: A Rejoinder
Valentine, J. C., Biglan, A., Boruch, R. F., Castro, F. G., Collins, L. M., Flay, B. R., Kellam, S. G., Mościcki, E. K., & Schinke, S. P. (2011) · Prevention Science · doi:10.1007/s11121-011-0220-y · Cited by 2 · see citing works
Replication Initiative: Prioritize Publication
Williams, S. N. (2012) · Science · doi:10.1126/science.336.6083.801-c · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A systematic review of learning computational thinking through Scratch in K-9
Zhang, L., & Nouri, J. (2019) · Computers & Education · doi:10.1016/j.compedu.2019.103607 · Cited by 452 · see citing works
Steenbergen-Hu, S., Makel, M. C., & Olszewski-Kubilius, P. (2016) · Review of Educational Research · doi:10.3102/0034654316675417 · Cited by 429 · see citing works
Self-citationAssessment rubrics: towards clearer and more replicable design, research and practice
Dawson, P. (2017) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2015.1111294 · Cited by 368 · see citing works
“Compared to other disciplines, educational researchers publish few replication studies (Makel and Plucker 2014).”
BackgroundHow school leadership influences student learning: A test of “The four paths model”
Leithwood, K., Sun, J., & Schumacker, R. (2020) · Educational Administration Quarterly · doi:10.1177/0013161x19878772 · Cited by 329 · see citing works
“DISCUSSION AND CONCLUSIONS Replication studies are relatively rare in the educational sciences, in general, and educational leadership studies, in particular ((Makel and Plucker, 2014).”
BackgroundChallenges for assessing replicability in preclinical cancer biology
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.67995 · Cited by 323 · see citing works
Soderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., Esterling, K. M., & Nosek, B. A. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01142-4 · Cited by 289 · see citing works
Marsden, E. J., Morgan‐Short, K., Thompson‐Lee, S., & Abugaber, D. (2018) · Language Learning · doi:10.1111/lang.12286 · Cited by 270 · see citing works
“To illustrate this with an example from the field of education research, Makel and Plucker (2014) found the median citation count of replications was five (range = 0 to 135), compared to 31 for the initial studies (range = 1 to 7,644).”
BackgroundMethodResultInfluentialRigorous Large-Scale Educational RCTs Are Often Uninformative: Should We Be Concerned?
Lortie-Forgues, H., & Inglis, M. (2019) · Educational Researcher · doi:10.3102/0013189x19832850 · Cited by 234 · see citing works
“This is worrying, as only 0.13% of articles in leading education journals report replication studies (Makel & Plucker, 2014).”
BackgroundInfluentialAdvances in nanomaterial application in enzyme-based electrochemical biosensors: a review
Kucherenko, I. S., Soldatkin, O. O., Kucherenko, D. Y., Soldatkina, O. V., & Dzyadevych, S. V. (2019) · Nanoscale Advances · doi:10.1039/c9na00491b · Cited by 232 · see citing works
What works and doesn't work with instructional video
Fiorella, L., & Mayer, R. E. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.07.015 · Cited by 212 · see citing works
Student success prediction in MOOCs
Gardner, J., & Brooks, C. (2018) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-018-9203-z · Cited by 210 · see citing works
“We note with some concern that there is a dearth of replication research in MOOCs in particular, and in the field of education in general (Makel and Plucker 2014).”
BackgroundMethodInfluentialEvaluating the Fairness of Predictive Student Models Through Slicing Analysis
Gardner, J., Brooks, C. A., & Baker, R. S. (2019) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3303772.3303791 · Cited by 193 · see citing works
“First, there has been only limited replication research in the field of learning analytics to date, with the replication rate (the percentage of studies which are replications) estimated at approximately 0.13% in the field of education [25].”
BackgroundInfluentialCanning, E. A., & Harackiewicz, J. M. (2015) · Motivation Science · doi:10.1037/mot0000015 · Cited by 195 · see citing works
Estimating the Reproducibility of Experimental Philosophy
Cova, F., Strickland, B., Abatista, A., Allard, A., Andow, J., Attie, M., Beebe, J. R., Berniûnas, R., Boudesseul, J., Colombo, M., et al. (2018) · Review of Philosophy and Psychology · doi:10.1007/s13164-018-0400-9 · Cited by 184 · see citing works
“…what the optimal replication rate1 should be for a given field in light of trade-offs between, e.g., innovation and confirmation (Gilbert et al. 2016; Makel & Plucker, 2014), many scientists regard the estimates that have been generated—less than 50% in each of the above cases—as worryingly low.”
BackgroundA Review of the Literature on Principal Turnover
Rangel, V. S. (2017) · Review of Educational Research · doi:10.3102/0034654317743197 · Cited by 178 · see citing works
“There often is little appetite to replicate existing work to investigate whether the posited relationships hold up under scrutiny and in different contexts (Makel & Plucker, 2014).”
BackgroundToh, W., & Kirschner, D. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.103912 · Cited by 170 · see citing works
Hall, C., Dahl‐Leonard, K., Cho, E., Solari, E. J., Capin, P., Conner, C. L., Henry, A. R., Cook, L., Hayes, L. L., Vargas, I., et al. (2022) · Reading Research Quarterly · doi:10.1002/rrq.477 · Cited by 169 · see citing works
Reconceptualizing replication as a sequence of different studies: A replication typology
Hüffmeier, J., Mazei, J., & Schultze, T. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.09.009 · Cited by 168 · see citing works
Replication in Criminology and the Social Sciences
Pridemore, W. A., Makel, M. C., & Plucker, J. A. (2017) · Annual Review of Criminology · doi:10.1146/annurev-criminol-032317-091849 · Cited by 152 · see citing works
Self-citationReplicated Evidence of Racial and Ethnic Disparities in Disability Identification in U.S. Schools
Morgan, P. L., Farkas, G., Hillemeier, M. M., & Maczuga, S. (2017) · Educational Researcher · doi:10.3102/0013189x17726282 · Cited by 145 · see citing works
“2Conceptual-type replications, in contrast to direct-type replication, provide for a more general test of the robustness of the findings of interest by using other samples, measures, and types of statistical analysis (Makel & Plucker, 2014).”
BackgroundMethodUnderstanding Science Achievement Gaps by Race/Ethnicity and Gender in Kindergarten and First Grade
Curran, F. C., & Kellogg, A. T. (2016) · Educational Researcher · doi:10.3102/0013189x16656611 · Cited by 136 · see citing works
“Additionally, by replicating the methodological approach utilized by Quinn and Cooc (2015), we respond to increased calls for replication in the educational research literature (Makel & Plucker, 2014).”
MethodThe Teaching and Learning Toolkit
Higgins, S. (2018) · Cambridge University Press eBooks · doi:10.1017/9781139519618.004 · Cited by 136 · see citing works
On the N th presence for the Community of Inquiry framework
Kozan, K., & Caskurlu, S. (2018) · Computers & Education · doi:10.1016/j.compedu.2018.03.010 · Cited by 139 · see citing works
Sáiz, M. S. I., Gómez, G. R., & Boud, D. J. (2020) · Higher Education · doi:10.1007/s10734-019-00469-2 · Cited by 152 · see citing works
Promoting Open Science to Increase the Trustworthiness of Evidence in Special Education
Cook, B. G., Lloyd, J., Mellor, D., Nosek, B. A., & Therrien, W. J. (2018) · The Exceptional Child · doi:10.1177/0014402918793138 · Cited by 115 · see citing works
“Consistent with findings from other fields (see Makel et al., 2012; Makel & Plucker, 2014), Makel et al. (2016) and Lemons et al. (2016) reported that only 0.5% and 0.4%, respectively, of articles published in special education were replication studies, and most of those were conceptual…”
ResultInfluentialSelf-citationFiorella, L., van Gog, T., Hoogerheide, V., & Mayer, R. E. (2016) · Journal of Educational Psychology · doi:10.1037/edu0000161 · Cited by 127 · see citing works
“This cross-national collaboration reflects the idea that replication is a crucial aspect of educational research (Makel & Plucker, 2014; Shavelson & Towne, 2002).”
Replication is important for educational psychology: Recent developments and key issues
Plucker, J. A., & Makel, M. C. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1895796 · Cited by 122 · see citing works
Self-citationSpillane, J. P., Hopkins, M., & Sweet, T. M. (2015) · American Journal of Education · doi:10.1086/683292 · Cited by 125 · see citing works
Replication Research, Publication Bias, and Applied Behavior Analysis
Tincani, M., & Travers, J. C. (2019) · Perspectives on Behavior Science · doi:10.1007/s40614-019-00191-5 · Cited by 126 · see citing works
Recommendations for Replication Research in Special Education
Coyne, M. D., Cook, B. G., & Therrien, W. J. (2016) · Remedial and Special Education · doi:10.1177/0741932516648463 · Cited by 117 · see citing works
“Indeed, replications sharing one or more common authors with the original studies are more likely to reproduce findings (Lemons et al., 2016; Makel & Plucker, 2014; Makel et al., 2016; Makel, Plucker, & Hegarty, 2012).”
BackgroundInfluentialReframing assessment research: through a practice perspective
Boud, D. J., Dawson, P., Bearman, M., Bennett, S., Gordon, J., & Molloy, E. (2016) · Studies in Higher Education · doi:10.1080/03075079.2016.1202913 · Cited by 116 · see citing works
“Recent findings about problems with replicability in educational research (Makel and Plucker 2014) may make the practice perspective proposed in this article attractive to researchers from a measurement perspective.”
Are Psychology Journals Anti-replication? A Snapshot of Editorial Practices
Martin, G., & Clarke, R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00523 · Cited by 106 · see citing works
“The situation is better in education (Makel and Plucker, 2014), human factors (Jones et al.”
BackgroundInfluentialObservational Evaluation of Teachers: Measuring More Than We Bargained for?
Campbell, S. L., & Ronfeldt, M. (2018) · American Educational Research Journal · doi:10.3102/0002831218776216 · Cited by 111 · see citing works
“In advocating for more replication studies in educational research, for example, Makel and Plucker (2014) revealed that less than 1% of education articles in top journals were replication studies.”
All the same or different? Revisiting measures of teachers' technology acceptance
Scherer, R., Siddiq, F., & Tondeur, J. (2019) · Computers & Education · doi:10.1016/j.compedu.2019.103656 · Cited by 133 · see citing works
van der Zee, T., & Reich, J. (2018) · AERA Open · doi:10.1177/2332858418787466 · Cited by 107 · see citing works
“These problems include high failure rates of replication studies (Makel & Plucker, 2014; Open Science Collaboration, 2012, 2015), publication bias (Rosenthal, 1979), high rates of false-positives (Ioannidis, 2005; Simmons, Nelson, & Simonsohn, 2011), and cost barriers to accessing scientific…”
BackgroundBest Practice Recommendations for Replicating Experiments in Public Administration
Walker, R. M., Brewer, G. A., Lee, M. J., Petrovsky, N., & van Witteloostuijn, A. C. (2018) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/muy047 · Cited by 105 · see citing works
Domain-specificity of self-regulated learning processing in science and history
Greene, J. A., Bolick, C. M., Jackson, W. P., Caprino, A. M., Oswald, C., & McVea, M. (2015) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2015.06.001 · Cited by 98 · see citing works
Replication of Special Education Research
Makel, M. C., Plucker, J. A., Freeman, J., Lombardi, A. R., Simonsen, B. M., & Coyne, M. D. (2016) · Remedial and Special Education · doi:10.1177/0741932516646083 · Cited by 97 · see citing works
“…higher (95% confidence interval [CI] = [3.12, 4.51]) than the replication rate of general education research, which had a replication rate of 0.13% (Makel & Plucker, 2014b), but less than half (95% CI = [0.41, 0.54]) the estimated replication rate of psychology journals (Makel et al., 2012), which…”
BackgroundMethodResultInfluentialSelf-citationBoth Questionable and Open Research Practices Are Prevalent in Education Research
Makel, M. C., Hodges, J., Cook, B. G., & Plucker, J. A. (2021) · Educational Researcher · doi:10.3102/0013189x211001356 · Cited by 92 · see citing works
“For example, to estimate how often replications were conducted in education research, Makel and Plucker (2014) reviewed the top 100 education journals as ranked by their impact factor.”
InfluentialSelf-citationSng, C. Y., Carter, M., & Stephenson, J. R. (2018) · Autism & Developmental Language Impairments · doi:10.1177/2396941518803806 · Cited by 93 · see citing works
Guerrero-Rosada, P., Weiland, C., McCormick, M. P., Hsueh, J., Sachs, J., Snow, C. E., & Maier, M. F. (2020) · Early Childhood Research Quarterly · doi:10.1016/j.ecresq.2020.07.009 · Cited by 94 · see citing works
Sex differences in the right tail of cognitive abilities: An update and cross cultural extension
Makel, M. C., Wai, J., Peairs, K. F., & Putallaz, M. (2016) · Intelligence · doi:10.1016/j.intell.2016.09.003 · Cited by 91 · see citing works
Self-citationWhen Lightning Strikes Twice: Profoundly Gifted, Profoundly Accomplished
Makel, M. C., Kell, H. J., Lubinski, D., Putallaz, M., & Benbow, C. P. (2016) · Psychological Science · doi:10.1177/0956797616644735 · Cited by 113 · see citing works
“Consequently, replication is essential (Makel & Plucker, 2014; Makel, Plucker, & Hegarty, 2012; Open Science Collaboration, 2015).”
Self-citationTwo Different Communication Genres and Implications for Vocabulary Development and Learning to Read
Massaro, D. W. (2015) · Journal of Literacy Research · doi:10.1177/1086296x15627528 · Cited by 81 · see citing works
“In the present constructive or conceptual replication (Makel & Plucker, 2014; Schmidt, 2009) and analysis, the written language sample consisted of the complete text of 112 picture books for a total of more than 57,000 words.”
BackgroundA discipline-wide investigation of the replicability of Psychology papers over the past two decades
Youyou, W., Yang, Y., & Uzzi, B. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2208863120 · Cited by 87 · see citing works
“To address key questions of replicability variation among subfields (22, 56), we parsed the overall distribution by subfield.”
Mitigating Illusory Results through Preregistration in Education
Gehlbach, H., & Robinson, C. D. (2017) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2017.1387950 · Cited by 79 · see citing works
“The scholarly community typically views replication attempts—even successful ones—as secondclass citizens due a host of different biases (Makel & Plucker, 2014).”
How Do Teachers Make Sense of Data in the Context of High-Stakes Decision Making?
Vanlommel, K., & Schildkamp, K. (2019) · American Educational Research Journal · doi:10.3102/0002831218803891 · Cited by 79 · see citing works
Kindergarten Black–White Test Score Gaps
Quinn, D. M. (2015) · Sociology of Education · doi:10.1177/0038040715573027 · Cited by 77 · see citing works
Wright, T. A., & Sweeney, D. A. (2015) · Journal of Organizational Behavior · doi:10.1002/job.2059 · Cited by 75 · see citing works
The role of replication studies in ecology
Fraser, H. S., Barnett, A., Parker, T. H., & Fidler, F. M. (2020) · Ecology and Evolution · doi:10.1002/ece3.6330 · Cited by 74 · see citing works
“Studies looking at the prevalence of self-identified “replication studies” in the literature find rates of 0.023% in ecology (Kelly, 2019), 0.1% in education (Makel & Plucker, 2014), and 1% in psychology (Makel, Plucker, & Hegarty, 2012).”
BackgroundInfluentialPublishing Single-Case Research Design Studies That Do Not Demonstrate Experimental Control
Tincani, M., & Travers, J. C. (2017) · Remedial and Special Education · doi:10.1177/0741932517697447 · Cited by 75 · see citing works
“Similarly, Makel and Plucker (2014) found that only .13% of articles published in the top 100 education journals were explicitly described as replications.”
BackgroundAutomated Text Data Mining Analysis of Five Decades of Educational Leadership Research Literature
Wang, Y.-Y., Bowers, A. J., & Fikis, D. R. J. (2017) · Educational Administration Quarterly · doi:10.1177/0013161x16660585 · Cited by 60 · see citing works
“It is thus pivotal for the field to strike a balance between being open to new ideas and the rigorous scrutiny of all ideas (Makel & Plucker, 2014; Sagan, 1997).”
BackgroundExpanding STEM opportunities through inclusive STEM‐focused high schools
Means, B., Wang, H., Wei, X., Lynch, S. J., Peters, V. L., Young, V. M., & Allen, C. D. (2017) · Science Education · doi:10.1002/sce.21281 · Cited by 79 · see citing works
A framework for developing and supporting student leadership in higher education
Skalicky, J., Pedersen, K. W., van der Meer, J., Fuglsang, S., Dawson, P., & Stewart, S. (2018) · Studies in Higher Education · doi:10.1080/03075079.2018.1522624 · Cited by 73 · see citing works
Physiology is hard: a replication study of students’ perceived learning difficulties
Slominski, T., Grindberg, S., & Momsen, J. L. (2019) · AJP Advances in Physiology Education · doi:10.1152/advan.00040.2018 · Cited by 73 · see citing works
Bernstein, B. O., Lubinski, D., & Benbow, C. P. (2020) · Journal of Educational Psychology · doi:10.1037/edu0000500 · Cited by 76 · see citing works
“Given the contemporary concern in the social sciences regarding the importance of replication (Camerer et al., 2018; Open Science Collaboration, 2015), conducting such appraisals of generalizability and robustness is essential (Makel & Plucker, 2014; Makel, Plucker, & Hegarty, 2012).”
BackgroundCharacteristics of effective leadership networks: a replication and extension
Leithwood, K. A. (2018) · School Leadership and Management · doi:10.1080/13632434.2018.1470503 · Cited by 71 · see citing works
“Replications studies are relatively rare in the educational sciences, in general, and educational leadership studies, in particular ((Makel and Plucker 2014).”
BackgroundA framework for learning from null results
Jacob, R. T., Doolittle, F., Kemple, J., & Somers, M.-A. (2019) · Educational Researcher · doi:10.3102/0013189x19891955 · Cited by 65 · see citing works
“Replication is important (Makel & Plucker, 2014), and in addition to thinking more carefully about the findings of individual studies, broader conclusions about an intervention’s effectiveness should be made by systematically reviewing and integrating findings based on multiple studies.”
BackgroundDawson, P., & Sutherland‐Smith, W. (2018) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2018.1531109 · Cited by 70 · see citing works
“No attempts to replicate these studies have been published, which although not uncommon in educational research (Makel and Plucker 2014), does present problems in terms of reliability.”
BackgroundHigher education: discipline or field of study?
Tight, M. (2020) · Tertiary Education and Management · doi:10.1007/s11233-020-09060-2 · Cited by 70 · see citing works
Relevance of Educational Research: An Ontological Conceptualization
Akkerman, S. F., Bakker, A., & Penuel, W. R. (2021) · Educational Researcher · doi:10.3102/0013189x211028239 · Cited by 70 · see citing works
“We see various criticisms, for example, that research findings are not considered useful, applicable, generalizable, or replicable (Broekkamp & van Hout-Wolters, 2007; Kim, 2019; Makel & Plucker, 2014; Vanderlinde & Van Braak, 2010).”
BackgroundCharalambous, C. Y., Hill, H. C., Chin, M. J., & McGinn, D. (2019) · Journal of Mathematics Teacher Education · doi:10.1007/s10857-019-09443-2 · Cited by 82 · see citing works
What constitutes high quality higher education pedagogical research?
Evans, C., Howson, C. K., Forsythe, A., & Edwards, C. (2020) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2020.1790500 · Cited by 68 · see citing works
“An analysis of the complete publication history of the current top 100 education journals ranked by 5-year impact factor, found that only 0.13% of education articles were replications (Makel and Plucker 2014).”
BackgroundInfluentialEducational technology: what it is and how it works
Dron, J. (2021) · AI & Society · doi:10.1007/s00146-021-01195-z · Cited by 68 · see citing works
Replication studies in international business
Dau, L. A., Santangelo, G. D., & van Witteloostuijn, A. C. (2021) · Journal of International Business Studies · doi:10.1057/s41267-021-00471-w · Cited by 68 · see citing works
Deaf Children’s ASL Vocabulary and ASL Syntax Knowledge Supports English Knowledge
Hoffmeister, R., Henner, J., Caldwell‐Harris, C. L., & Novogrodsky, R. (2021) · The Journal of Deaf Studies and Deaf Education · doi:10.1093/deafed/enab032 · Cited by 66 · see citing works
A framework for designing, implementing, communicating and researching peer assessment
Adachi, C., Tai, J., & Dawson, P. (2017) · Higher Education Research & Development · doi:10.1080/07294360.2017.1405913 · Cited by 65 · see citing works
“Our efforts presented here may contribute to addressing the ‘replication crisis’ in educational research (Makel & Plucker, 2014) by assisting future researchers to reproduce published peer assessment approaches.”
BackgroundBayesian statistics in educational research: a look at the current state of affairs
König, C., & van de Schoot, R. (2017) · Educational Review · doi:10.1080/00131911.2017.1350636 · Cited by 63 · see citing works
Lane, S. P., Steinley, D., & Sher, K. J. (2016) · Psychological Medicine · doi:10.1017/s0033291716000404 · Cited by 62 · see citing works
“This is especially critical given recent initiatives concerning the reproducibility of research findings across science as a whole (Nosek & Lakens, 2013; Collins & Tabak, 2014; Makel & Plucker, 2014).”
BackgroundPeters, S. J., Rambo-Hernandez, K. E., Makel, M. C., Matthews, M. S., & Plucker, J. A. (2017) · Gifted Child Quarterly · doi:10.1177/0016986217701834 · Cited by 62 · see citing works
“…using only one data set could have led to spurious findings due to idiosyncratic factors (e.g., whether it was Common Core aligned), our goals in selecting data sources were to triangulate data sources and to provide a type of internal self-replication of our findings (Makel & Plucker, 2014).”
MethodSelf-citationHorn, S., & Veermans, K. (2019) · Journal of Research in International Education · doi:10.1177/1475240919830003 · Cited by 69 · see citing works
Rate and success of study replication in ecology and evolution
Kelly, C. D. (2019) · PeerJ · doi:10.7717/peerj.7654 · Cited by 62 · see citing works
Novelty versus Replicability: Virtues and Vices in the Reward System of Science
Romero, F. (2017) · Philosophy of Science · doi:10.1086/694005 · Cited by 61 · see citing works
A Social Work Education Outcome Measure: The Evaluation Self-Efficacy Scale–II
Holden, G. W., Barker, K., Kuppens, S., & Rosenberg, G. (2017) · SAGE Open · doi:10.1177/2158244017728323 · Cited by 61 · see citing works
“The logical result is that calls for greater emphasis on, and more acceptance of, replication continue to be advanced across fields (e.g., Lindsay, 2015; Makel & Plucker, 2014; Warne, 2014).”
BackgroundGehlbach, H., & Robinson, C. D. (2021) · Educational Psychology · doi:10.1080/00461520.2021.1898961 · Cited by 46 · see citing works
“Makel and Plucker (2014) found that 0.13% of published educational research studies comprised replication attempts.”
BackgroundInfluentialDiPerna, J. C., Lei, P., Cheng, W., Hart, S. C., & Bellinger, J. M. (2017) · Journal of Educational Psychology · doi:10.1037/edu0000191 · Cited by 62 · see citing works
“As such, additional trials are necessary to determine if the SSIS-CIP effects replicate with new samples of first and second-grade students (e.g., Makel & Plucker, 2014).”
BackgroundUsing Standardized Test Scores to Include General Cognitive Ability in Education Research and Policy
Wai, J., Brown, M. I., & Chabris, C. F. (2018) · Journal of Intelligence · doi:10.3390/jintelligence6030037 · Cited by 59 · see citing works
“, [36–38]), with much focus on a lack of reproducibility and how to improve that, including specifically in education research [39].”
BackgroundKirksey, J. (2019) · AERA Open · doi:10.1177/2332858419867692 · Cited by 58 · see citing works
“…expectation that scholars should find statistically significant and large effect sizes to publish in top academic journals, the lack of attention and respect given to replication studies (Makel & Plucker, 2014), and the lack of transparency in how data are manipulated to produce reported results.”
BackgroundHensel, P. G. (2021) · European Management Journal · doi:10.1016/j.emj.2021.01.002 · Cited by 64 · see citing works
A decade of replication study in education? A mapping review (2011–2020)
Perry, T., Morris, R., & Lea, R. G. (2022) · Educational Research and Evaluation · doi:10.1080/13803611.2021.2022315 · Cited by 62 · see citing works
“…success rate was similar (67.6% vs. 67.4%), and – if we recombine our “approximate” and “conceptual” replication categories – we are in agreement with Makel and Plucker (2014) that approximately two thirds of replications are “conceptual/approximate” (as opposed to “direct”), that is, they…”
BackgroundMethodResultInfluentialBiomedical researchers’ perspectives on the reproducibility of research
Cobey, K. D., Ebrahimzadeh, S., Page, M. J., Thibault, R. T., Nguyen, P., Abu-Dalfa, F., & Moher, D. (2024) · PLoS Biology · doi:10.1371/journal.pbio.3002870 · Cited by 58 · see citing works
Heilporn, G., & Lakhal, S. (2019) · Computers & Education · doi:10.1016/j.compedu.2019.103712 · Cited by 60 · see citing works
Who Cares about Using Education Research in Policy and Practice?
OECD (2022) · Educational research and innovation · doi:10.1787/d7ff793d-en · Cited by 71 · see citing works
King, R. B., Haw, J. Y., & Wang, Y. (2024) · Learning and Instruction · doi:10.1016/j.learninstruc.2024.101978 · Cited by 71 · see citing works
“In other cases, the evidence base for educational recommendations is built on very small samples and might only work in highly contrived settings (e.g., Makel & Plucker, 2014).”
Loes, C. N., Salisbury, M. H., & Pascarella, E. T. (2014) · Higher Education · doi:10.1007/s10734-014-9807-0 · Cited by 56 · see citing works
Probleme evidenzbasierter bzw. -orientierter pädagogischer Praxis
Stark, R. (2017) · Zeitschrift für Pädagogische Psychologie · doi:10.1024/1010-0652/a000201 · Cited by 57 · see citing works
An a priori solution to the replication crisis
Trafimow, D. (2018) · Philosophical Psychology · doi:10.1080/09515089.2018.1490707 · Cited by 56 · see citing works
“…Asendorpf and colleagues (2013), Brandt and colleauges (2013), Earp (2016), Earp and Everett (2015), Earp et al. (2014), Earp and Trafimow (2015), Ioannidis (2005), Makel and Plucker (2014), McBee and Matthews (2014), Open Science Collaboration (2015), and Prinz, Schlange, and Asadullah (2011).”
BackgroundKlein, P., Küchemann, S., Brückner, S., Zlatkin‐Troitschanskaia, O., & Kühn, J. (2019) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.15.020116 · Cited by 57 · see citing works
Reproducibility in the Social Sciences
Moody, J. W., Keister, L. A., & Ramos, M. C. (2022) · Annual Review of Sociology · doi:10.1146/annurev-soc-090221-035954 · Cited by 55 · see citing works
Sandilos, L. E., Neugebauer, S. R., DiPerna, J. C., Hart, S. C., & Lei, P. (2022) · School Mental Health · doi:10.1007/s12310-022-09543-0 · Cited by 65 · see citing works
“Given the importance of replication in the educational sciences to build generalizable evidence (Makel & Plucker, 2014), this study builds on the findings of Sandilos et al.”
ResultEffects of performance feedback valence on perceptions of invested mental effort
Raaijmakers, S. F., Baars, M., Schaap, L., PAAS, F. G. W. C., & van Gog, T. (2016) · Learning and Instruction · doi:10.1016/j.learninstruc.2016.12.002 · Cited by 54 · see citing works
“…to directly replicate the fi ndings from Experiment 1 in Experiment 2, and then to conceptually replicate them in Experiment 3, is considered important for educational research (Bauernfeind, 1968; Makel & Plucker, 2014), and strengthens the theoretical as well as practical relevance of this study.”
Mapping the field of educational administration research: a journal citation network analysis
Wang, Y., & Bowers, A. J. (2016) · Journal of Educational Administration · doi:10.1108/jea-02-2015-0013 · Cited by 53 · see citing works
“In education research, Makel and Plucker (2014) cautioned against a value of novelty over truth in education sciences after noting that only 0.13% of education articles in the top 100 education journals ranked by 5-year impact factor were replications—the repetition of previous studies in order to…”
BackgroundInfluentialThe Role of Replication in Engineering Education Research
Benson, L., & Borrego, M. J. (2015) · Journal of Engineering Education · doi:10.1002/jee.20082 · Cited by 52 · see citing works
“They add confidence to findings (Spector et al., 2014), identify potential biases in an original study (Makel & Plucker, 2014), and provide a necessary basis for generalizability beyond the studied setting (Spector et al., 2014).”
ResultA Systematic Investigation of Replications in Computing Education Research
Hao, Q., Smith IV, D. H., Iriumi, N., Tsikerdekis, M., & Ko, A. J. (2019) · ACM Transactions on Computing Education · doi:10.1145/3345328 · Cited by 51 · see citing works
“Only journals with high impact factors are considered premium publication channels by most educational fields [10].”
BackgroundMethodInfluentialZinsser, K. M., Gordon, R. A., & Xue, J. (2021) · Early Childhood Research Quarterly · doi:10.1016/j.ecresq.2021.02.001 · Cited by 58 · see citing works
Baker, E., Williams, A. L., McLeod, S. L., & McCauley, R. J. (2018) · American Journal of Speech-Language Pathology · doi:10.1044/2018_ajslp-17-0127 · Cited by 52 · see citing works
A Replication by Any Other Name
Cook, B. G., Collins, L. W., Cook, S. C., & Cook, L. (2016) · Remedial and Special Education · doi:10.1177/0741932516637198 · Cited by 49 · see citing works
“Like many of the reviews in related fields (e.g., Makel & Plucker, 2014; Makel et al., 2012), Lemons et al. (2016, In this issue) and Makel et al. (2016, In this issue) identified replication studies using electronic searches for the keyword replicat* (i.e., variations of replicate).”
BackgroundResultInfluentialTesting the Efficacy of a Tier 2 Mathematics Intervention
Doabler, C. T., Clarke, B., Kosty, D. B., Kurtz‐Nelson, E., Fien, H., Smolkowski, K., & Baker, S. K. (2016) · Exceptional Children · doi:10.1177/0014402916660084 · Cited by 50 · see citing works
“The possibility of unintentional bias or potential repetition of mistakes in study logic or procedures because of author overlap is an additional limitation (Makel & Plucker, 2014).”
BackgroundDistributed Sensitivity to Syntax and Semantics throughout the Language Network
Shain, C., Kean, H., Casto, C., Lipkin, B., Affourtit, J., Siegelman, M., Mollica, F., & Fedorenko, E. G. (2024) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_02164 · Cited by 64 · see citing works
“…of a growing effort in the field to replicate influential findings (Gilmore et al., 2017; Yarkoni & Westfall, 2017; Open Science Collaboration, 2015; Makel & Plucker, 2014; Simons, 2014; Pashler & Wagenmakers, 2012), here, we revisit PDD ’ s claims across three fMRI experiments that address these…”
BackgroundMethodInfluentialA Concise Set of Core Recommendations to Improve the Dependability of Psychological Research
Lishner, D. A. (2015) · Review of General Psychology · doi:10.1037/gpr0000028 · Cited by 49 · see citing works
“The implication so far is that if one wishes to combat false findings, then one should rely on direct replication rather than conceptual replication (for a discussion of both replication types see Makel & Plucker, 2014; Schmidt, 2009).”
Background
- Authors
- Makel, M. C., & Plucker, J. A.
- Affiliations
- Duke University (US)
- University of Connecticut (US)
- Venue
- Educational Researcher
- Published
- 2014-08
- Volume
- 43
- Issue
- 6
- Pages
- 304-316
- ISSN
- 0013-189X (ISSN-L)
- 1935-102X
- OpenAlex
- W2139319950
- Semantic Scholar
- 493a2e93117a876871ac446b26acf47c50ea2cef
- Cited by
- 609
- References
- 131
- Citations by year
- 2026: 17 · 2025: 37 · 2024: 40 · 2023: 40 · 2022: 66 · 2021: 47 · 2020: 59 · 2019: 81 · 2018: 73 · 2017: 52 · 2016: 51 · 2015: 37 · 2014: 8
- FWCI
- 68.61
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Educational Researcher (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Educational Research and Methods
- Keywords
- replication, direct replication, educational research, publishing history, research credibility, methodological quality, replication success, educational periodicals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
