Outcome-Reporting Bias in Education Research

Pigott, Therese D.; Valentine, Jeffrey C.; Polanin, Joshua R.; Williams, Ryan T.; Canada, Dericka D.

doi:10.3102/0013189x13507104

Cited by 127Meta-analysis and systematic reviewsEducational Research and MethodsAcademic integrity and plagiarism

Outcome-Reporting Bias in Education Research

Open access

Pigott, Therese D., Valentine, Jeffrey C., Polanin, Joshua R., Williams, Ryan T., Canada, Dericka D.·2013·Educational Researcher

DOI 10.3102/0013189x13507104

Open this paper in CoChat — free account

Abstract

Outcome-reporting bias occurs when primary studies do not include information about all outcomes measured in a study. When studies omit findings on important measures, efforts to synthesize the research using systematic review techniques will be biased and interpretations of individual studies will be incomplete. Outcome-reporting bias has been well documented in medicine and has been shown to lead to inaccurate assessments of the effects of medical treatments and, in some cases, to omission of reports of harms. This study examines outcome-reporting bias in educational research by comparing the reports of educational interventions from dissertations to their published versions. We find that nonsignificant outcomes were 30% more likely to be omitted from a published study than statistically significant ones.

Integrity signals

Registry

Crossref · 10.3102/0013189x13507104

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Including unpublished studies in a systematic review reduces risk of publication bias.
“The inclusion of unpublished literature, in particular, is important to limit the risk of publication bias”
From Effects of after-school programs with at-risk youth on attendance and externalizing behaviors: A systematic review and meta-analysis (2015) · cites this paper as Pigott et al. 2013
After-school program intervention research may suffer from outcome reporting bias.
“We also suspect that outcome reporting bias may be an issue in after-school program intervention research as it has been found to be problematic in education research (Pigott et al. 2013).”
From Effects of after-school programs with at-risk youth on attendance and externalizing behaviors: A systematic review and meta-analysis (2015) · cites this paper as Pigott et al. 2013

CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-18 · Cited by 13,371 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
Schulz, K. F., Altman, D. G., Moher, D., & for the CONSORT Group (2010) · BMJ · doi:10.1136/bmj.c332 · Cited by 9,329 · see citing works
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · see citing works
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · see citing works
Selective publication of antidepressant trials and its influence on apparent efficacy
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa065779 · Cited by 2,435 · see citing works
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368 · see citing works
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · see citing works
Empirical Evidence for Selective Reporting of Outcomes in Randomized Trials: Comparison of Protocols to Published Articles
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · see citing works
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824 · see citing works
Positive Outcomes Influence the Rate and Time to Publication, but Not the Impact Factor of Publications of Clinical Trial Results
Suñé, P., Suñé, J. M., & Montoro, J. B. (2013) · PLoS ONE · doi:10.1371/journal.pone.0054583 · Cited by 1,687 · see citing works
How Many Studies Do You Need?: A Primer on Statistical Power for Meta-Analysis
Valentine, J. C., Pigott, T. D., & Rothstein, H. R. (2010) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998609346961 · Cited by 1,294 · see citing works
Self-citation
Publication bias in clinical trials due to statistical significance or direction of trial results
Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000006.pub3 · Cited by 898 · see citing works
Estimation of a Common Effect Parameter from Sparse Follow-Up Data
Greenland, S., & Robins, J. M. (1985) · Biometrics · doi:10.2307/2530643 · Cited by 798 · see citing works
Do studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748 · see citing works
Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors
Chan, A.-W., & Altman, D. G. (2005) · BMJ · doi:10.1136/bmj.38356.424606.8f · Cited by 613 · cited 7× · see citing works
“However, note that the Chan and Altman study found evidence of selective outcome reporting among outcomes labeled “important” – they just found more of it for outcomes that were perceived to be less important. Given the documented presence of outcome reporting bias in medical and social science research, we are intere…”
BackgroundResultInfluential
Reporting standards for research in psychology: why do we need them? What might they be?
APA Publications and Communications Board Working Group on Journal Article Reporting Standards (2008) · American Psychologist · doi:10.1037/0003-066x.63.9.839 · Cited by 571 · see citing works
Estimators of the Mantel-Haenszel Variance Consistent in Both Sparse Data and Large-Strata Limiting Models
Robins, J. M., Breslow, N. E., & Greenland, S. (1986) · Biometrics · doi:10.2307/2531052 · Cited by 476 · see citing works
Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research
Chan, A.-W. (2004) · Canadian Medical Association Journal · doi:10.1503/cmaj.1041086 · Cited by 444 · see citing works
Outcome Reporting in Industry-Sponsored Trials of Gabapentin for Off-Label Use
Vedula, S. S., Bero, L. A., Scherer, R. W., & Dickersin, K. (2009) · New England Journal of Medicine · doi:10.1056/nejmsa0906126 · Cited by 361 · see citing works
The relative benefits of meta-analysis conducted with individual participant data versus aggregated data
Cooper, H., & Patall, E. A. (2009) · Psychological Methods · doi:10.1037/a0015565 · Cited by 339 · see citing works
Statistical power of psychological research: What have we gained in 20 years?
Rossi, J. S. (1990) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.58.5.646 · Cited by 323 · see citing works
Statistical power of psychological research: What have we gained in 20 years?
Rossi, J. S. (1990) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.58.5.646 · Cited by 323 · see citing works
Rethinking and reframing the carnegie classification
McCormick, A. C., & Zhao, C. (2005) · Change The Magazine of Higher Learning · doi:10.3200/chng.37.5.51-57 · Cited by 240 · see citing works
Statistical Power and Effect Sizes of Clinical Neuropsychology Research
Bezeau, S., & Graves, R. E. (2001) · Journal of Clinical and Experimental Neuropsychology · doi:10.1076/jcen.23.3.399.1181 · Cited by 161 · see citing works
Finding the missing science: The fate of studies submitted for review by a human subjects committee.
Cooper, H., DeNeve, K., & Charlton, K. (1997) · Psychological Methods · doi:10.1037/1082-989x.2.4.447 · Cited by 144 · see citing works
Issues relating to confounding and meta‐analysis when including non‐randomized studies in systematic reviews on the effects of interventions
Valentine, J. C., & Thompson, S. G. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1064 · Cited by 130 · see citing works
Self-citation
Effects of deficient reporting on meta-analysis: A conceptual framework and reanalysis.
Orwin, R. G., & Cordray, D. S. (1985) · Psychological Bulletin · doi:10.1037/0033-2909.97.1.134 · Cited by 103 · see citing works
An Examination of the Precision and Technical Accuracy of the First Wave of Group-Randomized Trials Funded by the Institute of Education Sciences
Spybrook, J. K., & Raudenbush, S. W. (2009) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373709339524 · Cited by 88 · see citing works
Combining individual participant and aggregated data in a meta‐analysis with correlational studies
Pigott, T., Williams, R. T., & Polanin, J. R. (2012) · Research Synthesis Methods · doi:10.1002/jrsm.1051 · Cited by 27 · see citing works
Self-citation
How easily can omission of patients, or selection amongst poorly-reproducible measurements, create artificial correlations? Methods for detection and implications for observational research design in cardiology
Francis, D. P. (2012) · International Journal of Cardiology · doi:10.1016/j.ijcard.2011.12.018 · Cited by 23 · see citing works

Flagged first, then by citations

Sorted by citations

Facts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609 · see citing works
Does the Flipped Classroom Improve Student Learning and Satisfaction? A Systematic Review and Meta-Analysis
Låg, T., & Sæle, R. G. (2019) · AERA Open · doi:10.1177/2332858419870489 · Cited by 307 · see citing works
Estimating the Difference Between Published and Unpublished Effect Sizes: A Meta-Review
Polanin, J. R., Tanner-Smith, E. E., & Hennessy, E. A. (2016) · Review of Educational Research · doi:10.3102/0034654315582067 · Cited by 234 · see citing works
Self-citation
Editorial: Evidence on Questionable Research Practices: The Good, the Bad, and the Ugly
Banks, G. C., Rogelberg, S. G., Woznyj, H. M., Landis, R. S., & Rupp, D. E. (2016) · Journal of Business and Psychology · doi:10.1007/s10869-016-9456-7 · Cited by 176 · see citing works
Questions About Questionable Research Practices in the Field of Management: A Guest Commentary
Banks, G. C., O’Boyle, E. H., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., Abston, K. A., Bennett, A. A., & Adkins, C. L. (2016) · Journal of Management · doi:10.1177/0149206315619011 · Cited by 163 · see citing works
Replication in Criminology and the Social Sciences
Pridemore, W. A., Makel, M. C., & Plucker, J. A. (2017) · Annual Review of Criminology · doi:10.1146/annurev-criminol-032317-091849 · Cited by 152 · see citing works
Effects of after-school programs with at-risk youth on attendance and externalizing behaviors: A systematic review and meta-analysis
Kremer, K. P., Maynard, B. R., Polanin, J. R., Vaughn, M. G., & Sarteschi, C. M. (2015) · Journal of Youth and Adolescence · doi:10.1007/s10964-014-0226-4 · Cited by 132 · see citing works
Self-citation
Are Black Children Disproportionately Overrepresented in Special Education? A Best-Evidence Synthesis
Morgan, P. L., Farkas, G., Cook, M., Strassfeld, N. M., Hillemeier, M., Pun, W.-H., & Schussler, D. L. (2017) · Exceptional Children · doi:10.1177/0014402916664042 · Cited by 104 · see citing works
Critiques of Student Engagement
Macfarlane, B., & Tomlinson, M. (2017) · Higher Education Policy · doi:10.1057/s41307-016-0027-3 · Cited by 119 · see citing works
Enacting self-study as methodology for professional inquiry
Garbett, D., & Ovens, A. (2016) · Cited by 111 · see citing works
Life After NHST: How to Describe Your Data Without “p-ing” Everywhere
Valentine, J. C., Aloe, A. M., & Lau, T. S. (2015) · Basic and Applied Social Psychology · doi:10.1080/01973533.2015.1060240 · Cited by 103 · see citing works
Self-citation
Exclusionary School Discipline and Delinquent Outcomes: A Meta-Analysis
Gerlinger, J., Viano, S. L., Gardella, J. H., Fisher, B. W., Curran, F. C., & Higgins, E. M. (2021) · Journal of Youth and Adolescence · doi:10.1007/s10964-021-01459-3 · Cited by 101 · see citing works
Both Questionable and Open Research Practices Are Prevalent in Education Research
Makel, M. C., Hodges, J., Cook, B. G., & Plucker, J. A. (2021) · Educational Researcher · doi:10.3102/0013189x211001356 · Cited by 92 · see citing works
Improving transparency and reproducibility through registration: The status of intervention trials published in clinical psychology journals.
Cybulski, Ł., Mayo‐Wilson, E., & Grant, S. (2016) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000115 · Cited by 91 · see citing works
Meta-analyses of flipped classroom studies: A review of methodology
Hew, K. F., Bai, S., Dawson, P., & Lo, C. K. (2021) · Educational Research Review · doi:10.1016/j.edurev.2021.100393 · Cited by 90 · see citing works
Evidence-Based Management and the Trustworthiness of Our Cumulative Scientific Knowledge: Implications for Teaching, Research, and Practice
Kepes, S., Bennett, A. A., & McDaniel, M. A. (2014) · Academy of Management Learning and Education · doi:10.5465/amle.2013.0193 · Cited by 88 · see citing works
Actionable recommendations for narrowing the science-practice gap in open science
Aguinis, H., Banks, G. C., Rogelberg, S. G., & Cascio, W. F. (2020) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.02.007 · Cited by 81 · see citing works
A systematic literature review of Chinese music education studies during 2007 to 2019
Yang, Y., & Welch, G. (2023) · International Journal of Music Education · doi:10.1177/02557614221096150 · Cited by 108 · see citing works
Are Hispanic, Asian, Native American, or Language-Minority Children Overrepresented in Special Education?
Morgan, P. L., Farkas, G., Cook, M., Strassfeld, N. M., Hillemeier, M. M., Pun, W. H., Wang, Y., & Schussler, D. L. (2018) · Exceptional Children · doi:10.1177/0014402917748303 · Cited by 74 · see citing works
Preregistration in Single-Case Design Research
Johnson, A. H., & Cook, B. G. (2019) · Exceptional Children · doi:10.1177/0014402919868529 · Cited by 67 · see citing works
Questionable Research Practices in Experimental Communication Research: A Systematic Analysis From 1980 to 2013
Matthes, J., Marquart, F., Naderer, B., Arendt, F., Schmuck, D., & Adam, K. (2015) · Communication Methods and Measures · doi:10.1080/19312458.2015.1096334 · Cited by 66 · see citing works
Quality Indicators for Reviews of Research in Special Education
Talbott, E., Maggin, D. M., Acker, E. Y. V., & Kumm, S. (2017) · Exceptionality · doi:10.1080/09362835.2017.1283625 · Cited by 58 · see citing works
Three-Level Meta-Analysis of Single-Case Data Regarding the Effects of Peer Tutoring on Academic and Social-Behavioral Outcomes for At-Risk Students and Students With Disabilities
Moeyaert, M., Klingbeil, D. A., Rodabaugh, E., & Turan, M. (2019) · Remedial and Special Education · doi:10.1177/0741932519855079 · Cited by 53 · see citing works
Questionable research practices among researchers in the most research‐productive management programs
Kepes, S., Keener, S. K., McDaniel, M. A., & Hartman, N. S. (2022) · Journal of Organizational Behavior · doi:10.1002/job.2623 · Cited by 54 · see citing works
Sharing successes and hiding failures: ‘reporting bias’ in learning and teaching research
Dawson, P., & Dawson, S. L. (2016) · Studies in Higher Education · doi:10.1080/03075079.2016.1258052 · Cited by 48 · see citing works
On the use of flipped classroom across various disciplines: Insights from a second-order meta-analysis
Hew, K. F., Bai, S., Huang, W., Dawson, P., Du, J., Huang, G., Jia, C., & Khongjan, T. (2021) · Australasian Journal of Educational Technology · doi:10.14742/ajet.6475 · Cited by 59 · see citing works
Replication crisis or an opportunity to improve scientific production?
Navarro, M. D. F., Pascual‐Llobell, J., Pascual‐Soler, M., Perezgonzalez, J. D., & Berríos-Riquelme, J. (2020) · European Journal of Education · doi:10.1111/ejed.12417 · Cited by 46 · see citing works
The Effects of Co-Teaching and Related Collaborative Models of Instruction on Student Achievement: A Systematic Review and Meta-Analysis
Vembye, M. H., Weiss, F., & Bhat, B. H. (2023) · Review of Educational Research · doi:10.3102/00346543231186588 · Cited by 62 · see citing works
Vocational teachers' professional learning: A systematic literature review of the past decade
Zhou, N., Tigelaar, D. E. H., & Admiraal, W. (2022) · Teaching and Teacher Education · doi:10.1016/j.tate.2022.103856 · Cited by 39 · see citing works
Functional Assessment–Based Interventions for Students With or At-Risk for High-Incidence Disabilities: Field Testing Single-Case Synthesis Methods
Common, E. A., Lane, K. L., Pustejovsky, J. E., Johnson, A. H., & Johl, L. E. (2017) · Remedial and Special Education · doi:10.1177/0741932517693320 · Cited by 39 · see citing works
The Effects of One versus Two Years of Intensive Reading Intervention Implemented with Late Elementary Struggling Readers
Miciak, J., Roberts, G. J., Taylor, W. P., Solis, M. R., Ahmed, Y., Vaughn, S. R., & Fletcher, J. M. (2017) · Learning Disabilities Research and Practice · doi:10.1111/ldrp.12159 · Cited by 38 · see citing works
Creativity is more than novelty: Reconsidering replication as a creativity act.
Makel, M. C., & Plucker, J. (2014) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/a0035811 · Cited by 35 · see citing works
Transparent, Open, and Reproducible Prevention Science
Grant, S., Wendt, K. E., Leadbeater, B. J. R., Supplee, L. H., Mayo‐Wilson, E., Gardner, F. E. M., & Bradshaw, C. P. (2022) · Prevention Science · doi:10.1007/s11121-022-01336-w · Cited by 36 · see citing works
Implications of the open science era for educational psychology research syntheses
Patall, E. A. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1897009 · Cited by 34 · see citing works
Leadership training in healthcare: a systematic umbrella review
Phillipson, J., Pinto, A. C., Kingsley‐Smith, H., Krachler, N., McGivern, G., & Lyons, O. (2025) · BMJ Leader · doi:10.1136/leader-2025-001269 · Cited by 32 · see citing works
The Cognitive Profile of Math Difficulties: A Meta-Analysis Based on Clinical Criteria
Haberstroh, S., & Schulte‐Körne, G. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.842391 · Cited by 36 · see citing works
Rethinking ‘quantitative’ methods and the development of new researchers
Gorard, S. A. C. (2015) · Review of Education · doi:10.1002/rev3.3041 · Cited by 30 · see citing works
Student perspectives of independent and collaborative learning in a flipped foundational engineering course
Fox-Turnbull, W. H., & Docherty, P. D. (2018) · Australasian Journal of Educational Technology · doi:10.14742/ajet.3804 · Cited by 32 · see citing works
A Meta-analysis of the Effects of Mentoring on Youth in Foster Care
Poon, C. Y. S., Christensen, K. M., & Rhodes, J. E. (2021) · Journal of Youth and Adolescence · doi:10.1007/s10964-021-01472-6 · Cited by 29 · see citing works
Meta-analyzing the multiverse: A peek under the hood of selective reporting.
Olsson-Collentine, A., van Aert, R. C. M., Bakker, M., & Wicherts, J. M. (2023) · Psychological Methods · doi:10.1037/met0000559 · Cited by 34 · see citing works
Damaging Real Lives through Obstinacy: Re-Emphasising Why Significance Testing is Wrong
Gorard, S. (2016) · Sociological Research Online · doi:10.5153/sro.3857 · Cited by 26 · see citing works
EL META-ANÁLISIS: UNA METODOLOGÍA PARA LA INVESTIGACIÓN EN EDUCACIÓN
Botella, J., & Zamora, Á. (2017) · Educación XX1 · doi:10.5944/educxx1.19030 · Cited by 39 · see citing works
A retrospective analysis of dissemination biases in the brief alcohol intervention literature.
Tanner-Smith, E. E., & Polanin, J. R. (2015) · Psychology of Addictive Behaviors · doi:10.1037/adb0000014 · Cited by 26 · see citing works
“With few exceptions (McLeod & Weisz, 2004; Niemeyer, Musch, & Pietrowsky, 2013; Pigott et al., 2013), most empirical research on dissemination biases has been conducted in the fields of health and medicine.”
BackgroundResultSelf-citation
Functional Family Therapy for families of youth (age 11–18) with behaviour problems: A systematic review and meta-analysis
Littell, J. H., Pigott, T., Nilsen, W., Roberts, J., & Labrum, T. (2023) · Campbell Systematic Reviews · doi:10.1002/cl2.1324 · Cited by 22 · see citing works
Self-citation
Synthesizing Evidence in Public Policy Contexts
Valentine, J. C., Wilson, S. J., Rindskopf, D. M., Lau, T. S., Tanner‐Smith, E. E., Yeide, M., LaSota, R. R., & Foster, L. (2016) · Evaluation Review · doi:10.1177/0193841x16674421 · Cited by 22 · see citing works
Self-citation
Students With Emotional–Behavioral Disorders as Cross-Age Tutors: A Synthesis of the Literature
Watts, G. W., Bryant, D. P., & Carroll, M. L. (2018) · Behavioral Disorders · doi:10.1177/0198742918771914 · Cited by 20 · see citing works
Mathematics achievement and orientation: A systematic review and meta-analysis of education technology
Rakes, C. R., Ronau, R. N., Bush, S. B., Driskell, S. O. S., Niess, M. L., & Pugalee, D. (2020) · Educational Research Review · doi:10.1016/j.edurev.2020.100337 · Cited by 29 · see citing works
A Path to Greater Credibility: Large-Scale Collaborative Education Research
Makel, M. C., Smith, K. N., McBee, M. T., Peters, S. J., & Miller, E. M. (2019) · AERA Open · doi:10.1177/2332858419891963 · Cited by 25 · see citing works
A meta‐analysis of the effects of after‐school programmes among youth with marginalized identities
Christensen, K. M., Kremer, K. P., Poon, C. Y. S., & Rhodes, J. E. (2023) · Journal of Community & Applied Social Psychology · doi:10.1002/casp.2681 · Cited by 22 · see citing works
Unlocking the Potential of the “What Works” Approach to Policymaking and Practice
Granger, R. C., & Maynard, R. A. (2015) · American Journal of Evaluation · doi:10.1177/1098214015594420 · Cited by 19 · see citing works
Bestiary of Questionable Research Practices in Psychology
Nagy, T., Hergert, J., Elsherif, M. M., Wallrich, L., Schmidt, K., Waltzer, T., Payne, J. W., Gjoneska, B., Seetahul, Y., Wang, Y. A., et al. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251348431 · Cited by 24 · see citing works
Questionable Research Practices in Single-Case Experimental Designs: Examples and Possible Solutions
Tincani, M., & Travers, J. C. (2022) · Springer eBooks · doi:10.1007/978-3-031-04968-2_12 · Cited by 18 · see citing works
A Systematic Review of Peer Tutoring Interventions for Students with Disabilities
Talbott, E., Trzaska, A., & Zurheide, J. L. (2017) · doi:10.1002/9781118768778.ch16 · Cited by 18 · see citing works
Schulentwicklung in der digital geprägten Welt: Strategien, Rahmenbedingungen und Implikationen für Schulleitungshandeln
Waffner, B. (2021) · Waxmann Verlag GmbH eBooks · doi:10.31244/9783830994558.03 · Cited by 17 · see citing works
Effect of game-based learning on students’ mathematics high order thinking skills: A meta-analysis
Anggoro, B. S., Dewantara, A. H., Suherman, S., Muhammad, R. R., & Saraswati, S. (2024) · Revista de Psicodidáctica (English ed ) · doi:10.1016/j.psicoe.2024.500158 · Cited by 40 · see citing works
REES: A Registry of Efficacy and Effectiveness Studies in Education
Anderson, D., Spybrook, J. K., & Maynard, R. A. (2018) · Educational Researcher · doi:10.3102/0013189x18810513 · Cited by 15 · see citing works
Surfacing Inequities and Their Broader Implications in the CS Education Research Community
McGill, M. M., Davis, S., & Reyes, J. (2022) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3501385.3543969 · Cited by 14 · see citing works
Tempest in a teacup: An analysis of p-Hacking in organizational research
Gupta, A., & Bosco, F. A. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281938 · Cited by 15 · see citing works
Outcome Reporting Bias in Government-Sponsored Policy Evaluations: A Qualitative Content Analysis of 13 Studies
Vaganay, A. (2016) · PLoS ONE · doi:10.1371/journal.pone.0163702 · Cited by 15 · see citing works
Toward a Science of Failure Analysis: A Narrative Review
Allen‐Platt, C., Gerstner, C. E., Boruch, R. F., & Ruby, A. (2021) · Review of Research in Education · doi:10.3102/0091732x20985074 · Cited by 13 · see citing works
A Primer on Meta-Analytic Structural Equation Modeling: the Case of Depression
Valentine, J. C., Cheung, M. W. ‐., Smith, E. J., Alexander, O., Hatton, J. M., Hong, R. Y. S., Huckaby, L. T., Patton, S. C., Pössel, P., & Seely, H. D. (2021) · Prevention Science · doi:10.1007/s11121-021-01298-5 · Cited by 13 · see citing works
Self-citation
Questionable and Improved Research Practices in Single-Case Experimental Design: Initial Investigation and Findings
Tincani, M., Travers, J. C., Dowdy, A., Slocum, T. A., & Deitrich, R. (2025) · Perspectives on Behavior Science · doi:10.1007/s40614-025-00441-9 · Cited by 12 · see citing works
Positivity bias in higher education research
Tight, M. (2022) · Higher Education Quarterly · doi:10.1111/hequ.12388 · Cited by 12 · see citing works
Open Science Standards at Journals that Inform Evidence-Based Policy
Grant, S., Mayo‐Wilson, E., Kianersi, S., Naaman, K., & Henschel, B. (2023) · Prevention Science · doi:10.1007/s11121-023-01543-z · Cited by 12 · see citing works
Problem-based learning and student outcomes in higher education: a second-order meta-analysis
Erdem, C., Kaya, M., Toptaş, H. T., & Altunbaşak, İ. (2025) · Studies in Higher Education · doi:10.1080/03075079.2025.2498084 · Cited by 22 · see citing works
Preregistration of Randomized Controlled Trials
Cook, B. G., Wong, V. C., Fleming, J. I., & Solari, E. J. (2022) · Research on Social Work Practice · doi:10.1177/10497315221121117 · Cited by 11 · see citing works
Locating unregistered and unreported data for use in a social science systematic review and meta-analysis
Polanin, J. R., Espelage, D. L., Grotpeter, J. K., Valido, A., Ingram, K. M., Torgal, C., Sheikh, A. J. E., & Robinson, L. E. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01376-9 · Cited by 11 · see citing works
Self-citation
Realizing the promises of replication studies in education
Morrison, K. (2019) · Educational Research and Evaluation · doi:10.1080/13803611.2020.1838300 · Cited by 11 · see citing works
Die Bedeutung der Digitalisierung für das Management von Weiterbildungsorganisationen
Koschorreck, J., & Gundermann, A. (2021) · Waxmann Verlag GmbH eBooks · doi:10.31244/9783830994558.06 · Cited by 11 · see citing works
PROTOCOL: Assessment of outcome reporting bias in studies included in Campbell systematic reviews
Littell, J. H., Gorman, D. M., Valentine, J. C., & Pigott, T. (2023) · Campbell Systematic Reviews · doi:10.1002/cl2.1332 · Cited by 10 · see citing works
Self-citation
The Logic of Generalization From Systematic Reviews and Meta-Analyses of Impact Evaluations
Littell, J. H. (2024) · Evaluation Review · doi:10.1177/0193841x241227481 · Cited by 10 · see citing works
Editors’ Introduction: Randomized Controlled Trials Meet the Real World: The Nature and Consequences of Null Findings
Herrington, C. D., & Maynard, R. A. (2019) · Educational Researcher · doi:10.3102/0013189x19891441 · Cited by 10 · see citing works
Effects of Graphic Organizers on Outcomes for Students with Disabilities: Three-Level Meta-Analysis of Single-Case Studies
Urton, K., Moeyaert, M., Nobel, K., Barwasser, A., Boon, R. T., & Grünke, M. (2024) · Exceptionality · doi:10.1080/09362835.2024.2389080 · Cited by 13 · see citing works
Program and Policy Evaluations in Practice: Highlights from the Federal Perspective
Maynard, R. A., Goldstein, N. E. S., & Nightingale, D. S. (2016) · New Directions for Evaluation · doi:10.1002/ev.20209 · Cited by 9 · see citing works
The Role of Federal Agencies in Creating and Administering Evidence-Based Policies
Maynard, R. A. (2018) · The Annals of the American Academy of Political and Social Science · doi:10.1177/0002716218768742 · Cited by 9 · see citing works
Are U.S. Schools Discriminating When Suspending Students With Disabilities? A Best-Evidence Synthesis
Morgan, P. L., Wang, Y., Woods, A. D., Mandel, Z., Farkas, G., & Hillemeier, M. M. (2019) · Exceptional Children · doi:10.1177/0014402919868528 · Cited by 10 · see citing works
Reforms in Academic Publishing: Should Behavioral Disorders and Special Education Journals Embrace Them?
Cook, B. G. (2016) · Behavioral Disorders · doi:10.17988/0198-7429-41.3.161 · Cited by 9 · see citing works
Effects of selective outcome reporting on risk perception
Freudenstein, F., Croft, R. J., Loughran, S., Zeleke, B. M., & Wiedemann, P. M. (2021) · Environmental Research · doi:10.1016/j.envres.2021.110821 · Cited by 9 · see citing works
Guest Editor’s Introduction to Special Issue: The Science and Practice of Research Synthesis
Littell, J. H. (2013) · Journal of the Society for Social Work and Research · doi:10.5243/jsswr.2013.19 · Cited by 9 · see citing works
Guidance for Increasing Primary Study Inclusion and the Usability of Data in Meta-Analysis: A Reporting Tutorial
Chow, J. C., Sandbank, M. P., & Hampton, L. H. (2023) · Journal of Speech Language and Hearing Research · doi:10.1044/2023_jslhr-22-00318 · Cited by 8 · see citing works
A cautionary note on aggregation in educational psychology and beyond
Krammer, G. (2023) · Theory & Psychology · doi:10.1177/09593543231172495 · Cited by 7 · see citing works
Open Science 2.0: Large-Scale Collaborative Education ResearchPreprint
Makel, M. C., Smith, K. N., McBee, M. T., Peters, S. J., & Miller, E. M. (2019) · doi:10.31234/osf.io/ypmjg · Cited by 9 · see citing works
Digital gestützte Zusammenarbeit von Organisationen in der beruflichen Bildung
Hähn, K., & Niehoff, A. (2021) · Waxmann Verlag GmbH eBooks · doi:10.31244/9783830994558.05 · Cited by 7 · see citing works
The role of after-school programs in supporting social and emotional learning: A synthesis of systematic reviews
Jilink, L., Leseman, P. P. M., Slot, P. L., & Deynoot-Schaub, M. J. J. M. G. (2025) · International Journal of Educational Research Open · doi:10.1016/j.ijedro.2025.100546 · Cited by 11 · see citing works
Open Science als Beitrag zur Qualität in der Bildungsforschung
Krammer, G., & Svecnik, E. (2020) · Zeitschrift für Bildungsforschung · doi:10.1007/s35834-020-00286-z · Cited by 8 · see citing works
Evidence-informed practice in assessment: forging the nexus between research, policy & practice
Davari-Torshizi, M. (2020) · Assessment in Education Principles Policy and Practice · doi:10.1080/0969594x.2020.1815647 · Cited by 7 · see citing works
Academic achievement for students with and without disabilities in co-taught classrooms: A meta-analysis
Stefanidis, A., King‐Sears, M. E., Strogilos, V., Berkeley, S. L., DeLury, M., & Voulagka, A. (2023) · International Journal of Educational Research · doi:10.1016/j.ijer.2023.102208 · Cited by 9 · see citing works
Journal Editor and Associate Editor Perspectives on Research Reproducibility and Open Science
Maggin, D. M. (2021) · Remedial and Special Education · doi:10.1177/07419325211017294 · Cited by 6 · see citing works
How Effective Is Special Education? A Best-Evidence Synthesis
Gloski, C. A., Woods, A. D., Wang, Y., & Morgan, P. L. (2022) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-80117-494-720221008 · Cited by 6 · see citing works
Outcome Reporting Bias in Randomized Experiments on Substance Use Disorders
Wooditch, A., Sloas, L. B., Wu, X., & Key, A. (2018) · Journal of Quantitative Criminology · doi:10.1007/s10940-018-9388-7 · Cited by 6 · see citing works
Evaluating methodological quality of meta-analyses: A case study of meta-analyses on associations between parental involvement and students’ learning outcomes
Tan, C. Y., & Gao, L. (2025) · Educational Research Review · doi:10.1016/j.edurev.2025.100678 · Cited by 6 · see citing works
A quantitative analysis of the short-term and mid-term benefit of a flipped classroom for foundational engineering dynamics
Docherty, P. D., Zaka, P., & Fox-Turnbull, W. H. (2021) · Research Papers in Education · doi:10.1080/02671522.2020.1864773 · Cited by 5 · see citing works
The relationship between attributional style and symptoms of depression in youth: A systematic review and meta-analysis
Valentine, J. C., Pössel, P., Alharbi, A., Gerbine, S., Utterback, L., McClure, E. B., Mustapha, T. A., Nzama, N., Patel, K. S., Pratt, T. L., et al. (2024) · Journal of Social and Clinical Psychology · doi:10.1521/jscp.2024.43.4.377 · Cited by 4 · see citing works
Self-citation
A Registered Report on Selective Reporting in Single-Case Experimental Research
Travers, J. C., Tincani, M., Dowdy, A., Forbes, H. J., & Johnson, J. V. (2025) · Remedial and Special Education · doi:10.1177/07419325251350697 · Cited by 5 · see citing works
Outcome-Reporting Bias in Special Education Intervention Research Using Experimental and Quasi-Experimental Group Designs: A Conceptual Replication
Talbott, E., Maggin, D. M., Chua, M., Ashley, L., Chen, X., Chin, P., & Curry, M. K. (2024) · Remedial and Special Education · doi:10.1177/07419325241240067 · Cited by 4 · see citing works
The Role of Theory in Quantitative Data Analysis
Pigott, T. (2017) · SAGE Publications Ltd eBooks · doi:10.4135/9781473983953.n35 · Cited by 4 · see citing works
Self-citation
Reviews zur Organisationsentwicklung in Bildungseinrichtungen. Zur Methodik und Diskussion von Bias in Reviewverfahren der Bildungsforschung
Wilmers, A., Keller, C., & Anda, C. (2021) · Waxmann Verlag GmbH eBooks · doi:10.31244/9783830994558.01 · Cited by 3 · see citing works
Child-Focused Interventions to Promote the Academic and Social-Behavioral Competence of Rural Preschool Children: A Meta-Analysis
Smith, T. E., Stormont, M., Singell, E., McGreevy, M., & Reinke, W. (2024) · School psychology review · doi:10.1080/2372966x.2024.2399499 · Cited by 2 · see citing works
Meta-Analysis and Quality of Self-Monitoring Behavior Interventions with Children and Adolescents
Maag, J. W. (2019) · International Journal of Education · doi:10.5296/ije.v11i1.14580 · Cited by 3 · see citing works
Interventions Addressing Current Social Polarization in the School Context: A Scoping Review
Müller, M. Y., Santavicca, T., Miconi, D., & Rousseau, C. (2025) · The Journal of Early Adolescence · doi:10.1177/02724316251352126 · Cited by 3 · see citing works

Incomplete Reporting: Addressing the Problem of Outcome-Reporting Bias in Educational Research
Trainor, B. P. (2015) · Loyola eCommons (Loyola University of Chicago)
Incomplete Reporting: Addressing the Prevalence of Outcome-Reporting Bias in Educational Research.
Trainor, B. C., Polanin, J. R., Williams, R. T., & Pigott, T. (2015) · Society for Research on Educational Effectiveness
Self-citation
Outcome-Reporting Bias in Special Education Intervention Research Using Experimental and Quasi-Experimental Group Designs: A Conceptual Replication
Talbott, E., Maggin, D. M., Chua, M., Ashley, L., Chen, X., Chin, P., & Curry, M. K. (2024) · Remedial and Special Education · doi:10.1177/07419325241240067 · Cited by 4 · see citing works
Consequences of Outcome Reporting Bias in Education Research.
Williams, R. T., & Polanin, J. R. (2014) · Society for Research on Educational Effectiveness · Cited by 1 · see citing works
Self-citation
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · see citing works

Authors: Pigott, T. D., Valentine, J. C., Polanin, J. R., Williams, R. T., & Canada, D. D.
Affiliations: Loyola University Chicago, IL
University of Louisville (US)
Vanderbilt University (US)
University of Memphis (US)
Boston College (US)
ORCID: Valentine, Jeffrey C.
Polanin, Joshua R.
Williams, Ryan T.
Venue: Educational Researcher
Publisher: American Educational Research Association (AERA)
Published: 2013-11
Volume: 42
Issue: 8
Pages: 424-432
ISSN: 0013-189X (ISSN-L)
1935-102X
DOI: 10.3102/0013189x13507104
OpenAlex: W2122723488
Cited by: 127
References: 32
Citations by year: 2026: 2 · 2025: 12 · 2024: 11 · 2023: 12 · 2022: 12 · 2021: 23 · 2020: 7 · 2019: 11 · 2018: 7 · 2017: 8 · 2016: 10 · 2015: 7 · 2014: 4 · 2013: 1
FWCI: 7.98
Type: journal article
Language: en
Open access: green
ecommons.luc.edu
Licence: cc-by-nc-nd
journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Educational Researcher (publishedVersion)
Loyola eCommons (Loyola University of Chicago) (submittedVersion, cc-by-nc-nd)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Social Sciences › Education › Educational Research and Methods
Social Sciences › Safety Research › Academic integrity and plagiarism
Keywords: selective reporting, educational intervention, dissertations, publication bias, non-significant results, evidence synthesis, educational research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Outcome-Reporting Bias in Education Research

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks