The DOI matches its Crossref record; 132 of the 132 references it cites were found in a registry or an index.
Facts Are More Important Than Novelty: Replication in the Education Sciences
Abstract
Despite increased attention to methodological rigor in education research, the field has focused heavily on experimental design and not on the merit of replicating important results. The present study analyzed the complete publication history of the current top 100 education journals ranked by 5-year impact factor and found that only 0.13% of education articles were replications. Contrary to previous findings in medicine, but similar to psychology, the majority of education replications successfully replicated the original studies. However, replications were significantly less likely to be successful when there was no overlap in authorship between the original and replicating articles. The results emphasize the importance of third-party, direct replications in helping education research improve its ability to shape education policy and practice.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Novel or innovative results are more likely to be published than replications or incremental extensions of existing findings 13-15 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [15]
Citations
Cites
2 retracted
Wakefield, A., Murch, S., Anthony, A., Linnell, J., Casson, D., Malik, M., Berelowitz, M., Dhillon, A., Thomson, M., Harvey, P., et al. (1998) · The Lancet · doi:10.1016/s0140-6736(97)11096-0 · Cited by 3,006
RetractedLancet, T. (2010) · The Lancet · doi:10.1016/s0140-6736(10)60175-4 · Cited by 500
Retracted(1963) · Medical Entomology and Zoology · Cited by 15,085
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
An Introduction to Categorical Data Analysis
Agresti, A. (2006) · Wiley series in probability and statistics · doi:10.1002/0470114754 · Cited by 6,775
Journal of social behavior and personality
(1986) · Personality and Individual Differences · doi:10.1016/0191-8869(86)90025-5 · Cited by 6,751
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Moffitt, R. (2009) · American Economic Review · doi:10.1257/aer.99.2.660 · Cited by 4,888
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 5×
“Using health care research as an example, Ioannidis (2005a), in a review of highly cited medical publications (i.e., those cited more than 1,000 times), found only 44% of replications produced results similar to the original study.”
ResultInfluential
Research works printed but not linked: 14
- 10.1037//0003–066x.49.12.997 · DOI · Paper page
- Collins H. M., 1985, Changing order
- Kane E. J., 1984, Quarterly Journal of Business and Economics, 23, 3
- Neuliep J. W., 1993, Journal of Social Behavior and Personality, 8, 1
- Neuliep J. W., 1993, Journal of Social Behavior and Personality, 8, 21
- Perspectives on Psychological Science. (2012). Special section on replicability in psychological science: A crisis of confidence? Retrieved from http://pps.sagepub.com/content/7/6.toc · DOI · Paper page
- Powell K., 2007, PLoS Biology, 12, 2748
- Rhine J. B., 1997, Extra-sensory perception
- Ritchie S. J., 2012, Psychologist, 25, 346
- 10.1037//0033–2909.86.3.638 · DOI · Paper page
- Rotton J., 1995, Journal of Social Behavior and Personality, 10, 1
- Simons D. J., 2012, Psychologist, 25, 349
- Simons D. J., 2014, Perspectives on Psychological Science. APS Observer, 27, 16
- Williams S. N., 2012, Science, 336, 801
Cited by
Sorted by citations
A systematic review of learning computational thinking through Scratch in K-9
Zhang, L., & Nouri, J. (2019) · Computers & Education · doi:10.1016/j.compedu.2019.103607 · Cited by 538
Steenbergen-Hu, S., Makel, M. C., & Olszewski-Kubilius, P. (2016) · Review of Educational Research · doi:10.3102/0034654316675417 · Cited by 429
Self-citationAssessment rubrics: towards clearer and more replicable design, research and practice
Dawson, P. (2017) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2015.1111294 · Cited by 368
“Compared to other disciplines, educational researchers publish few replication studies (Makel and Plucker 2014).”
BackgroundHow school leadership influences student learning: A test of “The four paths model”
Leithwood, K., Sun, J., & Schumacker, R. (2020) · Educational Administration Quarterly · doi:10.1177/0013161x19878772 · Cited by 329
“DISCUSSION AND CONCLUSIONS Replication studies are relatively rare in the educational sciences, in general, and educational leadership studies, in particular ((Makel and Plucker, 2014).”
BackgroundChallenges for assessing replicability in preclinical cancer biology
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.67995 · Cited by 345
Soderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., Esterling, K. M., & Nosek, B. A. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01142-4 · Cited by 295
Marsden, E. J., Morgan‐Short, K., Thompson‐Lee, S., & Abugaber, D. (2018) · Language Learning · doi:10.1111/lang.12286 · Cited by 286
“To illustrate this with an example from the field of education research, Makel and Plucker (2014) found the median citation count of replications was five (range = 0 to 135), compared to 31 for the initial studies (range = 1 to 7,644).”
BackgroundMethodResultInfluentialRigorous Large-Scale Educational RCTs Are Often Uninformative: Should We Be Concerned?
Lortie-Forgues, H., & Inglis, M. (2019) · Educational Researcher · doi:10.3102/0013189x19832850 · Cited by 234
“This is worrying, as only 0.13% of articles in leading education journals report replication studies (Makel & Plucker, 2014).”
BackgroundInfluentialAdvances in nanomaterial application in enzyme-based electrochemical biosensors: a review
Kucherenko, I. S., Soldatkin, O. O., Kucherenko, D. Y., Soldatkina, O. V., & Dzyadevych, S. V. (2019) · Nanoscale Advances · doi:10.1039/c9na00491b · Cited by 235
What works and doesn't work with instructional video
Fiorella, L., & Mayer, R. E. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.07.015 · Cited by 272
Student success prediction in MOOCs
Gardner, J., & Brooks, C. (2018) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-018-9203-z · Cited by 210
“We note with some concern that there is a dearth of replication research in MOOCs in particular, and in the field of education in general (Makel and Plucker 2014).”
BackgroundMethodInfluentialEvaluating the Fairness of Predictive Student Models Through Slicing Analysis
Gardner, J., Brooks, C. A., & Baker, R. S. (2019) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3303772.3303791 · Cited by 229
“First, there has been only limited replication research in the field of learning analytics to date, with the replication rate (the percentage of studies which are replications) estimated at approximately 0.13% in the field of education [25].”
BackgroundInfluentialCanning, E. A., & Harackiewicz, J. M. (2015) · Motivation Science · doi:10.1037/mot0000015 · Cited by 211
Estimating the Reproducibility of Experimental Philosophy
Cova, F., Strickland, B., Abatista, A., Allard, A., Andow, J., Attie, M., Beebe, J. R., Berniûnas, R., Boudesseul, J., Colombo, M., et al. (2018) · Review of Philosophy and Psychology · doi:10.1007/s13164-018-0400-9 · Cited by 204
“…what the optimal replication rate1 should be for a given field in light of trade-offs between, e.g., innovation and confirmation (Gilbert et al. 2016; Makel & Plucker, 2014), many scientists regard the estimates that have been generated—less than 50% in each of the above cases—as worryingly low.”
BackgroundA Review of the Literature on Principal Turnover
Rangel, V. S. (2017) · Review of Educational Research · doi:10.3102/0034654317743197 · Cited by 178
“There often is little appetite to replicate existing work to investigate whether the posited relationships hold up under scrutiny and in different contexts (Makel & Plucker, 2014).”
BackgroundToh, W., & Kirschner, D. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.103912 · Cited by 170
Hall, C., Dahl‐Leonard, K., Cho, E., Solari, E. J., Capin, P., Conner, C. L., Henry, A. R., Cook, L., Hayes, L. L., Vargas, I., et al. (2022) · Reading Research Quarterly · doi:10.1002/rrq.477 · Cited by 258
Reconceptualizing replication as a sequence of different studies: A replication typology
Hüffmeier, J., Mazei, J., & Schultze, T. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.09.009 · Cited by 177
Replication in Criminology and the Social Sciences
Pridemore, W. A., Makel, M. C., & Plucker, J. A. (2017) · Annual Review of Criminology · doi:10.1146/annurev-criminol-032317-091849 · Cited by 158
Self-citationReplicated Evidence of Racial and Ethnic Disparities in Disability Identification in U.S. Schools
Morgan, P. L., Farkas, G., Hillemeier, M. M., & Maczuga, S. (2017) · Educational Researcher · doi:10.3102/0013189x17726282 · Cited by 145
“2Conceptual-type replications, in contrast to direct-type replication, provide for a more general test of the robustness of the findings of interest by using other samples, measures, and types of statistical analysis (Makel & Plucker, 2014).”
BackgroundMethodUnderstanding Science Achievement Gaps by Race/Ethnicity and Gender in Kindergarten and First Grade
Curran, F. C., & Kellogg, A. T. (2016) · Educational Researcher · doi:10.3102/0013189x16656611 · Cited by 136
“Additionally, by replicating the methodological approach utilized by Quinn and Cooc (2015), we respond to increased calls for replication in the educational research literature (Makel & Plucker, 2014).”
MethodThe Teaching and Learning Toolkit
Higgins, S. (2018) · Cambridge University Press eBooks · doi:10.1017/9781139519618.004 · Cited by 136
On the N th presence for the Community of Inquiry framework
Kozan, K., & Caskurlu, S. (2018) · Computers & Education · doi:10.1016/j.compedu.2018.03.010 · Cited by 139
Sáiz, M. S. I., Gómez, G. R., & Boud, D. J. (2020) · Higher Education · doi:10.1007/s10734-019-00469-2 · Cited by 178
Promoting Open Science to Increase the Trustworthiness of Evidence in Special Education
Adelson, J. L., Barton, E., Bradshaw, C., Nosek, B. A., Bryant, D., Cook, B. G., Coyne, M., DeBettencourt, L., DeHaven, A. C., Dymond, S., et al. (2018) · Exceptional Children · doi:10.1177/0014402918793138 · Cited by 136
“Consistent with findings from other fields (see Makel et al., 2012; Makel & Plucker, 2014), Makel et al. (2016) and Lemons et al. (2016) reported that only 0.5% and 0.4%, respectively, of articles published in special education were replication studies, and most of those were conceptual…”
ResultInfluentialSelf-citationFiorella, L., van Gog, T., Hoogerheide, V., & Mayer, R. E. (2016) · Journal of Educational Psychology · doi:10.1037/edu0000161 · Cited by 141
“This cross-national collaboration reflects the idea that replication is a crucial aspect of educational research (Makel & Plucker, 2014; Shavelson & Towne, 2002).”
Replication is important for educational psychology: Recent developments and key issues
Plucker, J. A., & Makel, M. C. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1895796 · Cited by 138
Self-citationSpillane, J. P., Hopkins, M., & Sweet, T. M. (2015) · American Journal of Education · doi:10.1086/683292 · Cited by 125
Replication Research, Publication Bias, and Applied Behavior Analysis
Tincani, M., & Travers, J. C. (2019) · Perspectives on Behavior Science · doi:10.1007/s40614-019-00191-5 · Cited by 136
Recommendations for Replication Research in Special Education
Coyne, M. D., Cook, B. G., & Therrien, W. J. (2016) · Remedial and Special Education · doi:10.1177/0741932516648463 · Cited by 120
“Indeed, replications sharing one or more common authors with the original studies are more likely to reproduce findings (Lemons et al., 2016; Makel & Plucker, 2014; Makel et al., 2016; Makel, Plucker, & Hegarty, 2012).”
BackgroundInfluentialReframing assessment research: through a practice perspective
Boud, D. J., Dawson, P., Bearman, M., Bennett, S., Gordon, J., & Molloy, E. (2016) · Studies in Higher Education · doi:10.1080/03075079.2016.1202913 · Cited by 116
“Recent findings about problems with replicability in educational research (Makel and Plucker 2014) may make the practice perspective proposed in this article attractive to researchers from a measurement perspective.”
Are Psychology Journals Anti-replication? A Snapshot of Editorial Practices
Martin, G., & Clarke, R. M. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00523 · Cited by 116
“The situation is better in education (Makel and Plucker, 2014), human factors (Jones et al.”
BackgroundInfluentialObservational Evaluation of Teachers: Measuring More Than We Bargained for?
Campbell, S. L., & Ronfeldt, M. (2018) · American Educational Research Journal · doi:10.3102/0002831218776216 · Cited by 111
“In advocating for more replication studies in educational research, for example, Makel and Plucker (2014) revealed that less than 1% of education articles in top journals were replication studies.”
All the same or different? Revisiting measures of teachers' technology acceptance
Scherer, R., Siddiq, F., & Tondeur, J. (2019) · Computers & Education · doi:10.1016/j.compedu.2019.103656 · Cited by 133
van der Zee, T., & Reich, J. (2018) · AERA Open · doi:10.1177/2332858418787466 · Cited by 122
“These problems include high failure rates of replication studies (Makel & Plucker, 2014; Open Science Collaboration, 2012, 2015), publication bias (Rosenthal, 1979), high rates of false-positives (Ioannidis, 2005; Simmons, Nelson, & Simonsohn, 2011), and cost barriers to accessing scientific…”
BackgroundBest Practice Recommendations for Replicating Experiments in Public Administration
Walker, R. M., Brewer, G. A., Lee, M. J., Petrovsky, N., & van Witteloostuijn, A. C. (2018) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/muy047 · Cited by 110
Domain-specificity of self-regulated learning processing in science and history
Greene, J. A., Bolick, C. M., Jackson, W. P., Caprino, A. M., Oswald, C., & McVea, M. (2015) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2015.06.001 · Cited by 110
Replication of Special Education Research
Makel, M. C., Plucker, J. A., Freeman, J., Lombardi, A. R., Simonsen, B. M., & Coyne, M. D. (2016) · Remedial and Special Education · doi:10.1177/0741932516646083 · Cited by 95
“…higher (95% confidence interval [CI] = [3.12, 4.51]) than the replication rate of general education research, which had a replication rate of 0.13% (Makel & Plucker, 2014b), but less than half (95% CI = [0.41, 0.54]) the estimated replication rate of psychology journals (Makel et al., 2012), which…”
BackgroundMethodResultInfluentialSelf-citationBoth Questionable and Open Research Practices Are Prevalent in Education Research
Makel, M. C., Hodges, J., Cook, B. G., & Plucker, J. A. (2021) · Educational Researcher · doi:10.3102/0013189x211001356 · Cited by 97
“For example, to estimate how often replications were conducted in education research, Makel and Plucker (2014) reviewed the top 100 education journals as ranked by their impact factor.”
InfluentialSelf-citationSng, C. Y., Carter, M., & Stephenson, J. R. (2018) · Autism & Developmental Language Impairments · doi:10.1177/2396941518803806 · Cited by 99
Guerrero-Rosada, P., Weiland, C., McCormick, M. P., Hsueh, J., Sachs, J., Snow, C. E., & Maier, M. F. (2020) · Early Childhood Research Quarterly · doi:10.1016/j.ecresq.2020.07.009 · Cited by 104
Sex differences in the right tail of cognitive abilities: An update and cross cultural extension
Makel, M. C., Wai, J., Peairs, K. F., & Putallaz, M. (2016) · Intelligence · doi:10.1016/j.intell.2016.09.003 · Cited by 93
Self-citationWhen Lightning Strikes Twice: Profoundly Gifted, Profoundly Accomplished
Makel, M. C., Kell, H. J., Lubinski, D., Putallaz, M., & Benbow, C. P. (2016) · Psychological Science · doi:10.1177/0956797616644735 · Cited by 113
“Consequently, replication is essential (Makel & Plucker, 2014; Makel, Plucker, & Hegarty, 2012; Open Science Collaboration, 2015).”
Self-citationTwo Different Communication Genres and Implications for Vocabulary Development and Learning to Read
Massaro, D. W. (2015) · Journal of Literacy Research · doi:10.1177/1086296x15627528 · Cited by 83
“In the present constructive or conceptual replication (Makel & Plucker, 2014; Schmidt, 2009) and analysis, the written language sample consisted of the complete text of 112 picture books for a total of more than 57,000 words.”
BackgroundA discipline-wide investigation of the replicability of Psychology papers over the past two decades
Youyou, W., Yang, Y., & Uzzi, B. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2208863120 · Cited by 99
“To address key questions of replicability variation among subfields (22, 56), we parsed the overall distribution by subfield.”
Mitigating Illusory Results through Preregistration in Education
Gehlbach, H., & Robinson, C. D. (2017) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2017.1387950 · Cited by 82
“The scholarly community typically views replication attempts—even successful ones—as secondclass citizens due a host of different biases (Makel & Plucker, 2014).”
How Do Teachers Make Sense of Data in the Context of High-Stakes Decision Making?
Vanlommel, K., & Schildkamp, K. (2019) · American Educational Research Journal · doi:10.3102/0002831218803891 · Cited by 79
Kindergarten Black–White Test Score Gaps
Quinn, D. M. (2015) · Sociology of Education · doi:10.1177/0038040715573027 · Cited by 89
Wright, T. A., & Sweeney, D. A. (2015) · Journal of Organizational Behavior · doi:10.1002/job.2059 · Cited by 75
The role of replication studies in ecology
Fraser, H. S., Barnett, A., Parker, T. H., & Fidler, F. M. (2020) · Ecology and Evolution · doi:10.1002/ece3.6330 · Cited by 74
“Studies looking at the prevalence of self-identified “replication studies” in the literature find rates of 0.023% in ecology (Kelly, 2019), 0.1% in education (Makel & Plucker, 2014), and 1% in psychology (Makel, Plucker, & Hegarty, 2012).”
BackgroundInfluentialPublishing Single-Case Research Design Studies That Do Not Demonstrate Experimental Control
Tincani, M., & Travers, J. C. (2017) · Remedial and Special Education · doi:10.1177/0741932517697447 · Cited by 82
“Similarly, Makel and Plucker (2014) found that only .13% of articles published in the top 100 education journals were explicitly described as replications.”
BackgroundAutomated Text Data Mining Analysis of Five Decades of Educational Leadership Research Literature
Wang, Y., Bowers, A. J., & Fikis, D. J. (2016) · Educational Administration Quarterly · doi:10.1177/0013161x16660585 · Cited by 79
“It is thus pivotal for the field to strike a balance between being open to new ideas and the rigorous scrutiny of all ideas (Makel & Plucker, 2014; Sagan, 1997).”
BackgroundExpanding STEM opportunities through inclusive STEM‐focused high schools
Means, B., Wang, H., Wei, X., Lynch, S. J., Peters, V. L., Young, V. M., & Allen, C. D. (2017) · Science Education · doi:10.1002/sce.21281 · Cited by 79
A framework for developing and supporting student leadership in higher education
Skalicky, J., Pedersen, K. W., van der Meer, J., Fuglsang, S., Dawson, P., & Stewart, S. (2018) · Studies in Higher Education · doi:10.1080/03075079.2018.1522624 · Cited by 73
Physiology is hard: a replication study of students’ perceived learning difficulties
Slominski, T., Grindberg, S., & Momsen, J. L. (2019) · AJP Advances in Physiology Education · doi:10.1152/advan.00040.2018 · Cited by 75
Bernstein, B. O., Lubinski, D., & Benbow, C. P. (2020) · Journal of Educational Psychology · doi:10.1037/edu0000500 · Cited by 76
“Given the contemporary concern in the social sciences regarding the importance of replication (Camerer et al., 2018; Open Science Collaboration, 2015), conducting such appraisals of generalizability and robustness is essential (Makel & Plucker, 2014; Makel, Plucker, & Hegarty, 2012).”
BackgroundCharacteristics of effective leadership networks: a replication and extension
Leithwood, K. A. (2018) · School Leadership and Management · doi:10.1080/13632434.2018.1470503 · Cited by 91
“Replications studies are relatively rare in the educational sciences, in general, and educational leadership studies, in particular ((Makel and Plucker 2014).”
BackgroundDawson, P., & Sutherland‐Smith, W. (2018) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2018.1531109 · Cited by 70
“No attempts to replicate these studies have been published, which although not uncommon in educational research (Makel and Plucker 2014), does present problems in terms of reliability.”
BackgroundHigher education: discipline or field of study?
Tight, M. (2020) · Tertiary Education and Management · doi:10.1007/s11233-020-09060-2 · Cited by 70
Relevance of Educational Research: An Ontological Conceptualization
Akkerman, S. F., Bakker, A., & Penuel, W. R. (2021) · Educational Researcher · doi:10.3102/0013189x211028239 · Cited by 99
“We see various criticisms, for example, that research findings are not considered useful, applicable, generalizable, or replicable (Broekkamp & van Hout-Wolters, 2007; Kim, 2019; Makel & Plucker, 2014; Vanderlinde & Van Braak, 2010).”
BackgroundCharalambous, C. Y., Hill, H. C., Chin, M. J., & McGinn, D. (2019) · Journal of Mathematics Teacher Education · doi:10.1007/s10857-019-09443-2 · Cited by 82
What constitutes high quality higher education pedagogical research?
Evans, C., Howson, C. K., Forsythe, A., & Edwards, C. (2020) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2020.1790500 · Cited by 98
“An analysis of the complete publication history of the current top 100 education journals ranked by 5-year impact factor, found that only 0.13% of education articles were replications (Makel and Plucker 2014).”
BackgroundInfluentialEducational technology: what it is and how it works
Dron, J. (2021) · AI & Society · doi:10.1007/s00146-021-01195-z · Cited by 68
Replication studies in international business
Dau, L. A., Santangelo, G. D., & van Witteloostuijn, A. C. (2021) · Journal of International Business Studies · doi:10.1057/s41267-021-00471-w · Cited by 74
Deaf Children’s ASL Vocabulary and ASL Syntax Knowledge Supports English Knowledge
Hoffmeister, R., Henner, J., Caldwell‐Harris, C. L., & Novogrodsky, R. (2021) · The Journal of Deaf Studies and Deaf Education · doi:10.1093/deafed/enab032 · Cited by 66
A framework for learning from null results
Jacob, R. T., Doolittle, F., Kemple, J., & Somers, M.-A. (2019) · Educational Researcher · doi:10.3102/0013189x19891955 · Cited by 65
“Replication is important (Makel & Plucker, 2014), and in addition to thinking more carefully about the findings of individual studies, broader conclusions about an intervention’s effectiveness should be made by systematically reviewing and integrating findings based on multiple studies.”
BackgroundA framework for designing, implementing, communicating and researching peer assessment
Adachi, C., Tai, J., & Dawson, P. (2017) · Higher Education Research & Development · doi:10.1080/07294360.2017.1405913 · Cited by 74
“Our efforts presented here may contribute to addressing the ‘replication crisis’ in educational research (Makel & Plucker, 2014) by assisting future researchers to reproduce published peer assessment approaches.”
BackgroundBayesian statistics in educational research: a look at the current state of affairs
König, C., & van de Schoot, R. (2017) · Educational Review · doi:10.1080/00131911.2017.1350636 · Cited by 63
Lane, S. P., Steinley, D., & Sher, K. J. (2016) · Psychological Medicine · doi:10.1017/s0033291716000404 · Cited by 61
“This is especially critical given recent initiatives concerning the reproducibility of research findings across science as a whole (Nosek & Lakens, 2013; Collins & Tabak, 2014; Makel & Plucker, 2014).”
BackgroundPeters, S. J., Rambo-Hernandez, K. E., Makel, M. C., Matthews, M. S., & Plucker, J. A. (2017) · Gifted Child Quarterly · doi:10.1177/0016986217701834 · Cited by 62
“…using only one data set could have led to spurious findings due to idiosyncratic factors (e.g., whether it was Common Core aligned), our goals in selecting data sources were to triangulate data sources and to provide a type of internal self-replication of our findings (Makel & Plucker, 2014).”
MethodSelf-citationHorn, S., & Veermans, K. (2019) · Journal of Research in International Education · doi:10.1177/1475240919830003 · Cited by 83
Rate and success of study replication in ecology and evolution
Kelly, C. D. (2019) · PeerJ · doi:10.7717/peerj.7654 · Cited by 60
Novelty versus Replicability: Virtues and Vices in the Reward System of Science
Romero, F. (2017) · Philosophy of Science · doi:10.1086/694005 · Cited by 61
A Social Work Education Outcome Measure: The Evaluation Self-Efficacy Scale–II
Holden, G. W., Barker, K., Kuppens, S., & Rosenberg, G. (2017) · SAGE Open · doi:10.1177/2158244017728323 · Cited by 61
“The logical result is that calls for greater emphasis on, and more acceptance of, replication continue to be advanced across fields (e.g., Lindsay, 2015; Makel & Plucker, 2014; Warne, 2014).”
BackgroundGehlbach, H., & Robinson, C. D. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1898961 · Cited by 62
“Makel and Plucker (2014) found that 0.13% of published educational research studies comprised replication attempts.”
BackgroundInfluentialDiPerna, J. C., Lei, P., Cheng, W., Hart, S. C., & Bellinger, J. M. (2017) · Journal of Educational Psychology · doi:10.1037/edu0000191 · Cited by 62
“As such, additional trials are necessary to determine if the SSIS-CIP effects replicate with new samples of first and second-grade students (e.g., Makel & Plucker, 2014).”
BackgroundUsing Standardized Test Scores to Include General Cognitive Ability in Education Research and Policy
Wai, J., Brown, M. I., & Chabris, C. F. (2018) · Journal of Intelligence · doi:10.3390/jintelligence6030037 · Cited by 59
“, [36–38]), with much focus on a lack of reproducibility and how to improve that, including specifically in education research [39].”
BackgroundKirksey, J. J. (2019) · AERA Open · doi:10.1177/2332858419867692 · Cited by 69
“…expectation that scholars should find statistically significant and large effect sizes to publish in top academic journals, the lack of attention and respect given to replication studies (Makel & Plucker, 2014), and the lack of transparency in how data are manipulated to produce reported results.”
BackgroundHensel, P. G. (2021) · European Management Journal · doi:10.1016/j.emj.2021.01.002 · Cited by 73
A decade of replication study in education? A mapping review (2011–2020)
Perry, T., Morris, R., & Lea, R. G. (2022) · Educational Research and Evaluation · doi:10.1080/13803611.2021.2022315 · Cited by 62
“…success rate was similar (67.6% vs. 67.4%), and – if we recombine our “approximate” and “conceptual” replication categories – we are in agreement with Makel and Plucker (2014) that approximately two thirds of replications are “conceptual/approximate” (as opposed to “direct”), that is, they…”
BackgroundMethodResultInfluentialBiomedical researchers’ perspectives on the reproducibility of research
Cobey, K. D., Ebrahimzadeh, S., Page, M. J., Thibault, R. T., Nguyen, P., Abu-Dalfa, F., & Moher, D. (2024) · PLoS Biology · doi:10.1371/journal.pbio.3002870 · Cited by 80
Heilporn, G., & Lakhal, S. (2019) · Computers & Education · doi:10.1016/j.compedu.2019.103712 · Cited by 60
Who Cares about Using Education Research in Policy and Practice?
OECD (2022) · Educational research and innovation · doi:10.1787/d7ff793d-en · Cited by 78
King, R. B., Haw, J. Y., & Wang, Y. (2024) · Learning and Instruction · doi:10.1016/j.learninstruc.2024.101978 · Cited by 71
“In other cases, the evidence base for educational recommendations is built on very small samples and might only work in highly contrived settings (e.g., Makel & Plucker, 2014).”
Loes, C. N., Salisbury, M. H., & Pascarella, E. T. (2014) · Higher Education · doi:10.1007/s10734-014-9807-0 · Cited by 67
Probleme evidenzbasierter bzw. -orientierter pädagogischer Praxis
Stark, R. (2017) · Zeitschrift für Pädagogische Psychologie · doi:10.1024/1010-0652/a000201 · Cited by 64
An a priori solution to the replication crisis
Trafimow, D. (2018) · Philosophical Psychology · doi:10.1080/09515089.2018.1490707 · Cited by 56
“…Asendorpf and colleagues (2013), Brandt and colleauges (2013), Earp (2016), Earp and Everett (2015), Earp et al. (2014), Earp and Trafimow (2015), Ioannidis (2005), Makel and Plucker (2014), McBee and Matthews (2014), Open Science Collaboration (2015), and Prinz, Schlange, and Asadullah (2011).”
BackgroundKlein, P., Küchemann, S., Brückner, S., Zlatkin‐Troitschanskaia, O., & Kühn, J. (2019) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.15.020116 · Cited by 59
Reproducibility in the Social Sciences
Moody, J. W., Keister, L. A., & Ramos, M. C. (2022) · Annual Review of Sociology · doi:10.1146/annurev-soc-090221-035954 · Cited by 63
Sandilos, L. E., Neugebauer, S. R., DiPerna, J. C., Hart, S. C., & Lei, P. (2022) · School Mental Health · doi:10.1007/s12310-022-09543-0 · Cited by 84
“Given the importance of replication in the educational sciences to build generalizable evidence (Makel & Plucker, 2014), this study builds on the findings of Sandilos et al.”
ResultEffects of performance feedback valence on perceptions of invested mental effort
Raaijmakers, S. F., Baars, M., Schaap, L., PAAS, F. G. W. C., & van Gog, T. (2016) · Learning and Instruction · doi:10.1016/j.learninstruc.2016.12.002 · Cited by 67
“…to directly replicate the fi ndings from Experiment 1 in Experiment 2, and then to conceptually replicate them in Experiment 3, is considered important for educational research (Bauernfeind, 1968; Makel & Plucker, 2014), and strengthens the theoretical as well as practical relevance of this study.”
Mapping the field of educational administration research: a journal citation network analysis
Wang, Y., & Bowers, A. J. (2016) · Journal of Educational Administration · doi:10.1108/jea-02-2015-0013 · Cited by 53
“In education research, Makel and Plucker (2014) cautioned against a value of novelty over truth in education sciences after noting that only 0.13% of education articles in the top 100 education journals ranked by 5-year impact factor were replications—the repetition of previous studies in order to…”
BackgroundInfluentialThe Role of Replication in Engineering Education Research
Benson, L., & Borrego, M. J. (2015) · Journal of Engineering Education · doi:10.1002/jee.20082 · Cited by 52
“They add confidence to findings (Spector et al., 2014), identify potential biases in an original study (Makel & Plucker, 2014), and provide a necessary basis for generalizability beyond the studied setting (Spector et al., 2014).”
ResultA Systematic Investigation of Replications in Computing Education Research
Hao, Q., Smith IV, D. H., Iriumi, N., Tsikerdekis, M., & Ko, A. J. (2019) · ACM Transactions on Computing Education · doi:10.1145/3345328 · Cited by 51
“Only journals with high impact factors are considered premium publication channels by most educational fields [10].”
BackgroundMethodInfluentialZinsser, K. M., Gordon, R. A., & Xue, J. (2021) · Early Childhood Research Quarterly · doi:10.1016/j.ecresq.2021.02.001 · Cited by 70
Baker, E., Williams, A. L., McLeod, S. L., & McCauley, R. J. (2018) · American Journal of Speech-Language Pathology · doi:10.1044/2018_ajslp-17-0127 · Cited by 52
A Replication by Any Other Name
Cook, B. G., Collins, L. W., Cook, S. C., & Cook, L. (2016) · Remedial and Special Education · doi:10.1177/0741932516637198 · Cited by 52
“Like many of the reviews in related fields (e.g., Makel & Plucker, 2014; Makel et al., 2012), Lemons et al. (2016, In this issue) and Makel et al. (2016, In this issue) identified replication studies using electronic searches for the keyword replicat* (i.e., variations of replicate).”
BackgroundResultInfluentialTesting the Efficacy of a Tier 2 Mathematics Intervention
Doabler, C. T., Clarke, B., Kosty, D. B., Kurtz‐Nelson, E., Fien, H., Smolkowski, K., & Baker, S. K. (2016) · Exceptional Children · doi:10.1177/0014402916660084 · Cited by 50
“The possibility of unintentional bias or potential repetition of mistakes in study logic or procedures because of author overlap is an additional limitation (Makel & Plucker, 2014).”
BackgroundDistributed Sensitivity to Syntax and Semantics throughout the Language Network
Shain, C., Kean, H., Casto, C., Lipkin, B., Affourtit, J., Siegelman, M., Mollica, F., & Fedorenko, E. G. (2024) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_02164 · Cited by 69
“…of a growing effort in the field to replicate influential findings (Gilmore et al., 2017; Yarkoni & Westfall, 2017; Open Science Collaboration, 2015; Makel & Plucker, 2014; Simons, 2014; Pashler & Wagenmakers, 2012), here, we revisit PDD ’ s claims across three fMRI experiments that address these…”
BackgroundMethodInfluentialA Concise Set of Core Recommendations to Improve the Dependability of Psychological Research
Lishner, D. A. (2015) · Review of General Psychology · doi:10.1037/gpr0000028 · Cited by 49
“The implication so far is that if one wishes to combat false findings, then one should rely on direct replication rather than conceptual replication (for a discussion of both replication types see Makel & Plucker, 2014; Schmidt, 2009).”
Background
Metadata
- Authors
- Makel, M. C., & Plucker, J. A.
- Affiliations
- Duke University (US)
- University of Connecticut (US)
- Venue
- Educational Researcher
- Published
- 2014-08
- Volume
- 43
- Issue
- 6
- Pages
- 304-316
- ISSN
- 0013-189X (ISSN-L)
- 1935-102X
- OpenAlex
- W2139319950
- Semantic Scholar
- 493a2e93117a876871ac446b26acf47c50ea2cef
- Cited by
- 609
- References
- 131
- Citations by year
- 2026: 17 · 2025: 37 · 2024: 40 · 2023: 40 · 2022: 66 · 2021: 47 · 2020: 59 · 2019: 81 · 2018: 73 · 2017: 52 · 2016: 51 · 2015: 37 · 2014: 8
- FWCI
- 68.61
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Educational Researcher (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Educational Research and Methods
- Keywords
- replication, direct replication, educational research, publishing history, research credibility, methodological quality, replication success, educational periodicals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
