The DOI matches its Crossref record; 36 of the 37 references it cites were found in a registry or an index.
Outlier Removal and the Relation with Reporting Errors and Quality of Psychological Research
Open accessAbstract
BACKGROUND: The removal of outliers to acquire a significant result is a questionable research practice that appears to be commonly used in psychology. In this study, we investigated whether the removal of outliers in psychology papers is related to weaker evidence (against the null hypothesis of no effect), a higher prevalence of reporting errors, and smaller sample sizes in these papers compared to papers in the same journals that did not report the exclusion of outliers from the analyses. METHODS AND FINDINGS: We retrieved a total of 2667 statistical results of null hypothesis significance tests from 153 articles in main psychology journals, and compared results from articles in which outliers were removed (N = 92) with results from articles that reported no exclusion of outliers (N = 61). We preregistered our hypotheses and methods and analyzed the data at the level of articles. Results show no significant difference between the two types of articles in median p value, sample sizes, or prevalence of all reporting errors, large reporting errors, and reporting errors that concerned the statistical significance. However, we did find a discrepancy between the reported degrees of freedom of t tests and the reported sample size in 41% of articles that did not report removal of any data values. This suggests common failure to report data exclusions (or missingness) in psychological articles. CONCLUSIONS: We failed to find that the removal of outliers from the analysis in psychological articles was related to weaker evidence (against the null hypothesis of no effect), sample size, or the prevalence of errors. However, our control sample might be contaminated due to nondisclosure of excluded values in articles that did not report exclusion of outliers. Results therefore highlight the importance of more transparent reporting of statistical analyses.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, there is evidence that many reported p-values do not match their accompanying test statistic and degrees of freedom (Bakker & Wicherts, 2011; Bakker & Wicherts, 2014; Berle & Starcevic, 2007; Caperos & Pardo, 2013; Garcia-Berthou & Alcaraz, 2004; Veldkamp, Nuijten, Dominguez-Alvarez, Van Assen, & Wicherts, 2014; Wicherts, Bakker, & Molenaar, 2011).”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Bakker & Wicherts, 2014
The cited source says what the claim attributes to it.
In the citing paper
“Bakker and Wicherts (2014) only included completely reported t and F values.”
From The prevalence of statistical reporting errors in psychology (1985–2013) (2016) · cites this paper as Bakker and Wicherts (2014)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Methods for Research Workers
Fisher, R. A. (1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,883
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Publication Manual of the American Psychological Association
DeCleene, K. E., & Fogo, J. (2011) · Occupational Therapy In Health Care · doi:10.3109/07380577.2011.629024 · Cited by 3,178
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
The power of outliers (and why researchers should ALWAYS check for them)
Osborne, J. W., & Overbay, A. (2020) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.7275/qf69-7k43 · Cited by 898
Sharing Detailed Research Data Is Associated with Increased Citation Rate
Piwowar, H. A., Day, R. S., & Fridsma, D. B. (2007) · PLoS ONE · doi:10.1371/journal.pone.0000308 · Cited by 881
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M. T., Fruyt, F. D., Houwer, J. D., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R. M., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 876
Self-citationThe rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800
Self-citationSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2012) · doi:10.2139/ssrn.2160588 · Cited by 680
The poor availability of psychological research data for reanalysis.
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006) · American Psychologist · doi:10.1037/0003-066x.61.7.726 · Cited by 543
Self-citationStatistical Methods for Research Workers.
Jowett, G. H., & Fisher, R. A. (1956) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.2307/2985747 · Cited by 515
Data Withholding in Academic Genetics
Campbell, E. G., Clarridge, B. R., Gokhale, M., Birenbaum, L., Hilgartner, S., Holtzman, N. A., & Blumenthal, D. (2002) · JAMA · doi:10.1001/jama.287.4.473 · Cited by 491
The Availability of Research Data Declines Rapidly with Article Age
Vines, T. H., Albert, A. Y. K., Andrew, R. L., Débarre, F., Bock, D. G., Franklin, M. T., Gilbert, K. J., Moore, J.-S., Renaut, S., & Rennison, D. J. (2014) · Current Biology · doi:10.1016/j.cub.2013.11.014 · Cited by 482
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026828 · Cited by 424
Self-citationEmpirical Study of Data Sharing by Authors Publishing in PLoS Journals
Savage, C. J., & Vickers, A. J. (2009) · PLoS ONE · doi:10.1371/journal.pone.0007078 · Cited by 411
The (mis)reporting of statistical results in psychology journals
Bakker, M., & Wicherts, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0089-5 · Cited by 356
Self-citationEich, E. (2014) · Psychology Science · doi:10.1177/0956797613512465 · Cited by 267
Meta‐research: The art of getting it wrong
Ioannidis, J. P. A. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.19 · Cited by 237
Wolins, L. (1962) · American Psychologist · doi:10.1037/h0038819 · Cited by 231
Selective reporting biases in cancer prognostic factor studies.
Kyzas, P., Loizou, K. T., & Ioannidis, J. (2005) · Journal of the National Cancer Institute · doi:10.1093/jnci/dji184 · Cited by 226
Modern Statistics for the Social and Behavioral Sciences: A Practical Introduction
Wilcox, R. R. (2011) · Cited by 220
Verifying the Solution from a Nonlinear Solver: A Case Study
McCullough, B. D., & Vinod, H. D. (2003) · American Economic Review · doi:10.1257/000282803322157133 · Cited by 220
Replication Standards for Quantitative Social Science
Jeremy Freese, A. (2007) · Sociological Methods & Research · doi:10.1177/0049124107306659 · Cited by 201
Bakker, M., & Wicherts, J. (2014) · Psychological methods · doi:10.1037/met0000014 · Cited by 173
Self-citationLeBel, E. P., Borsboom, D., Giner‐Sorolla, R., Hasselman, F., Peters, K. R., Ratliff, K. A., & Smith, C. T. (2013) · Perspectives on Psychological Science · doi:10.1177/1745691613491437 · Cited by 106
The Life of p : “Just Significant” Results are on the Rise
Leggett, N. C., Thomas, N. A., Loetscher, T., & Nicholls, M. E. R. (2013) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2013.863371 · Cited by 64
Data Sharing in Medical Research: An Empirical Investigation
Reidpath, D. D., & Allotey, P. A. (2001) · Bioethics · doi:10.1111/1467-8519.00220 · Cited by 63
Modern Statistics for the Social and Behavioral Sciences
Wilcox, R. (2011) · doi:10.1201/9781466503236 · Cited by 62
(2014) · Data Archiving and Networked Services (DANS) · Cited by 46
Consistency errors in p-values reported in Spanish psychology journals
Caperos, J., & Pardo, A. (2013) · Psicothema · doi:10.7334/psicothema2012.207 · Cited by 23
Replication Data Sets and Favored-Hypothesis Bias
Firebaugh, G. (2007) · Sociological Methods & Research · doi:10.1177/0049124107306663 · Cited by 19
How Often are Our Statistics Wrong? a Statistics Class Exercise
Rossi, J. S. (1987) · Teaching of Psychology · doi:10.1207/s15328023top1402_8 · Cited by 15
When ab ≠ c – c′: Published errors in the reports of single-mediator models
Petrocelli, J. V., Clarkson, J. J., Whitmire, M. B., & Moon, P. E. (2012) · Behavior Research Methods · doi:10.3758/s13428-012-0262-5 · Cited by 12
Research works printed but not linked: 2
- Molenaar D, Wicherts JM, Borsboom D (2005) Participant loss in psychological research and the problem of exclusion: A validity issue. [Bachelor's thesis]. Amsterdam: University of Amsterdam.
- Wicherts JM (2008) Expectancy bias, statistical blunders, and the importance of reanalysis. Invited talk at the 23 IOPS summer conference, June 19, Enschede, The Netherlands.
Cited by
Sorted by citations
The prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510
“…there is evidence that many reported p -values do not match their accompanying test statistic and degrees of freedom (Bakker & Wicherts, 2011; Bakker & Wicherts, 2014; Berle & Starcevic, 2007; Caperos & Pardo, 2013; Garcia-Berthou & Alcaraz, 2004; Veldkamp, Nuijten, Dominguez-Alvarez, Van…”
Self-citationSample Size, Statistical Power, and False Conclusions in Infant Looking‐Time Research
Oakes, L. M. (2017) · Infancy · doi:10.1111/infa.12186 · Cited by 269
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616650874 · Cited by 219
Self-citationEditorial: Evidence on Questionable Research Practices: The Good, the Bad, and the Ugly
Banks, G. C., Rogelberg, S. G., Woznyj, H. M., Landis, R. S., & Rupp, D. E. (2016) · Journal of Business and Psychology · doi:10.1007/s10869-016-9456-7 · Cited by 176
Questions About Questionable Research Practices in the Field of Management: A Guest Commentary
Banks, G. C., O’Boyle, E. H., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., Abston, K. A., Bennett, A. A., & Adkins, C. L. (2016) · Journal of Management · doi:10.1177/0149206315619011 · Cited by 163
Metadata
- Authors
- Bakker, M., & Wicherts, J. M.
- Affiliations
- University of Amsterdam (NL)
- Tilburg University (NL)
- Corresponding author
- Bakker, Marjan
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2014-07-29
- Volume
- 9
- Issue
- 7
- Pages
- e103360
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4114807
- OpenAlex
- W2012571365
- Cited by
- 101
- References
- 37
- Citations by year
- 2026: 5 · 2025: 9 · 2024: 16 · 2023: 13 · 2022: 6 · 2021: 8 · 2020: 9 · 2019: 3 · 2018: 9 · 2017: 8 · 2016: 6 · 2015: 7 · 2014: 2
- FWCI
- 7.51
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- Research portal (Tilburg University) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (016-125-385)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Education › Academic integrity and plagiarism
- Keywords
- outlier removal, questionable research practices, error reporting, null hypothesis significance testing, sample size, research quality, psychology journals, data exclusion
- MeSH terms
- Humans, Psychology, Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
