Analytic Review as a Solution to the Misreporting of Statistical Results in Psychological Science
Published version of a preprint
The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
In this article, we propose analytic review (AR) as a solution to the problem of misreporting statistical results in psychological science. AR requires authors submitting manuscripts for publication to also submit the data file and syntax used during analyses. Regular reviewers or statistical experts then review reported analyses in order to verify that the analyses reported were actually conducted and that the statistical values are accurately reported. We begin by describing the problem of misreporting in psychology and introduce the basic AR process. We then highlight both primary and secondary benefits of adopting AR and describe different permutations of the AR system, each of which has its own strengths and limitations. We conclude by attempting to dispel three anticipated concerns about AR: that it will increase the workload placed on scholars, that it will infringe on the traditional peer-review process, and that it will hurt the image of the discipline of psychology. Although implementing AR will add one more step to the bureaucratic publication process, we believe it can be implemented in an efficient manner that would greatly assist in decreasing the frequency and impact of misreporting while also providing secondary benefits in other domains of scientific integrity.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Some journals, such as the American Journal of Political Science, have already gone a step further, and employ a third-party organization to check the analytic reproducibility of all articles before publishing them [47].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [47]
Citations
Cites
2 retracted
RETRACTED: Syncope in Adults With Pulmonary Arterial Hypertension
Le, R. J., Fenstad, E. R., Kremers, H. M., McCully, R. B., Frantz, R. P., McGoon, M. D., & Kane, G. C. (2011) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2011.04.026 · Cited by 30
Retracted(2012) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2012.04.006 · Cited by 1
RetractedThe file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
(1997) · Obesity Surgery · doi:10.1381/096089297765555359 · Cited by 3,916
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Publication Manual of the American Psychological Association
DeCleene, K. E., & Fogo, J. (2012) · Occupational Therapy In Health Care · doi:10.3109/07380577.2011.629024 · Cited by 3,167
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · cited 1×
“Investigations into the frequency of misreporting statistical values have yielded varying estimates of the prevalence of misreporting (e.g., 1.0%–2.0% in Fanelli, 2009; 22.0%–23.3% in John, Loewenstein, & Prelec, 2012).”
BackgroundInfluentialP-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
Hoffman, S. G., & Smits, J. A. J. (2008) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.v69n0415 · Cited by 1,420
Gender differences in scholastic achievement: A meta-analysis.
Voyer, D., & Voyer, S. D. (2014) · Psychological Bulletin · doi:10.1037/a0036620 · Cited by 1,337 · cited 1×
“…Burke, Martens, & Faucher, 2010, the effectiveness of a particular therapy, Hofmann & Smits, 2008, or gender differences in scholastic achievement, Voyer & Voyer, 2014), AR could focus on the important statistics related to primary statistical comparison of interest as opposed to reviewing…”
BackgroundNosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff
Herndon, T., Ash, M., & Pollin, R. (2014) · Cambridge Journal of Economics · doi:10.1093/cje/bet075 · Cited by 1,197
Two Decades of Terror Management Theory: A Meta-Analysis of Mortality Salience Research
Burke, B. L., Martens, A., & Faucher, E. H. (2010) · Personality and Social Psychology Review · doi:10.1177/1088868309352321 · Cited by 1,188
Eta Squared, Partial Eta Squared, and Misreporting of Effect Size in Communication Research
Levine, T. R., & Hullett, C. R. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00828.x · Cited by 998
A meta-analytic review of research on gender differences in sexuality, 1993–2007.
Petersen, J. L., & Hyde, J. S. (2010) · Psychological Bulletin · doi:10.1037/a0017504 · Cited by 931
Bem, D. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021524 · Cited by 893 · cited 1×
“Under such an approach, Bem’s (2011) infamous paper on precognition would likely be a more suitable candidate for AR in comparison with a paper examining a less extraordinary claim.”
BackgroundThe poor availability of psychological research data for reanalysis.
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006) · American Psychologist · doi:10.1037/0003-066x.61.7.726 · Cited by 543
Promoting Transparency in Social Science Research
Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., Glennerster, R., Green, D. P., Humphreys, M., Imbens, G., et al. (2014) · Science · doi:10.1126/science.1245317 · Cited by 540
Brand, J. L. (1997) · Digital Commons - Andrews University (Andrews University) · Cited by 537
Reinhart, C. M., & Rogoff, K. S. (2010) · American Economic Review · doi:10.1257/aer.100.2.573 · Cited by 438
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026828 · Cited by 424 · cited 7×
“Adopting this more purposive approach to selecting analyses or articles for AR would better facilitate addressing the problem of innocent or deliberate misreporting, as authors are most likely to misreport statistics from analyses related to their central claims (Wicherts et al., 2011).”
BackgroundInfluentialScientific Utopia: I. Opening Scientific Communication
Nosek, B. A., & Bar-Anan, Y. (2012) · Psychological Inquiry · doi:10.1080/1047840x.2012.692215 · Cited by 368
The (mis)reporting of statistical results in psychology journals
Bakker, M., & Wicherts, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0089-5 · Cited by 356 · cited 1×
“Bakker and Wicherts (2011) recently adopted a different approach: sampling articles to study misreporting.”
BackgroundMcNutt, M. (2014) · Science · doi:10.1126/science.1250475 · Cited by 291
Funder, D. C., Levine, J. M., Mackie, D. M., Morf, C. C., Sansone, C., Vazire, S., & West, S. G. (2014) · Personality and Social Psychology Review · doi:10.1177/1088868313507536 · Cited by 245
Just Post It: The Lesson From Two Cases of Fabricated Data Detected by Statistics Alone
Simonsohn, U. (2013) · Psychological Science · doi:10.1177/0956797613480366 · Cited by 216
Statistical Reform in Psychology: Is Anything Changing?
Cumming, G., Fidler, F., Leonard, M., Kalinowski, P., Christiansen, A., Kleinig, A., Lo, J., McMenamin, N., & Wilson, S. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.01881.x · Cited by 122
Crocker, J., & Cooper, M. L. (2011) · Science · doi:10.1126/science.1216775 · Cited by 84
Nelson, L. D., Simmons, J. P., & Simonsohn, U. (2012) · Psychological Inquiry · doi:10.1080/1047840x.2012.705245 · Cited by 78
Toward Improved Statistical Reporting in the Journal of Consulting and Clinical Psychology.
Fidler, F., Cumming, G., Thomason, N., Pannuzzo, D., Smith, J., Fyffe, P., Edmonds, H., Harrington, C., & Schmitt, R. (2005) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.73.1.136 · Cited by 72
The Review Process at PSPB: Correlates of Interreviewer Agreement and Manuscript Acceptance
Petty, R. E., Fleming, M. A., & Fabrigar, L. R. (1999) · Personality and Social Psychology Bulletin · doi:10.1177/0146167299025002005 · Cited by 61
Feduzi, A., Runde, J., & Zappia, C. (2013) · Cambridge Journal of Economics · doi:10.1093/cje/bet054 · Cited by 53
Psychologists Are Open to Change, yet Wary of Rules
Fuchs, H. M., Jenny, M., & Fiedler, S. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459521 · Cited by 50
European Bulletin of Social Psychology
Classen, S., de Lemus, S., Iatridis, T., Muldoon, O. T., Naughton, C., & Walsh, S. (2012) · Cited by 43
P-Curve and Effect Size: Correcting for Publication Bias Using Only Significant ResultsPreprint
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2377290 · Cited by 22
Research works printed but not linked: 2
- Herndon T., 2013, Cambridge Journal of Economics, 38, 1
- Simonsohn U., Journal of Experimental Psychology: General
Cited by
Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917747646 · Cited by 784
The prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510
“This solution would be in line with the notion of Analytic Review (Sakaluk, Williams, & Biernat, 2014), in which a reviewer receives the data file and syntax of a manuscript to check if the reported analyses were actually conducted and reported correctly.”
InfluentialHardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Henry Tessler, M., et al. (2018) · Royal Society Open Science · doi:10.1098/rsos.180448 · Cited by 336
Scientific Utopia III: Crowdsourcing Science
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850561 · Cited by 174
Hardwicke, T. E., Bohn, M., MacDonald, K., Hembacher, E., Nuijten, M. B., Peloquin, B. N., deMayo, B. E., Long, B., Yoon, E. J., & Frank, M. C. (2021) · Royal Society Open Science · doi:10.1098/rsos.201494 · Cited by 108
“Journals could also conduct independent assessment of analytic reproducibility prior to publication, as has been adopted by the American Journal of Political Science [22]; however, this would naturally require additional resources (for discussion see [23]).”
Method
Metadata
- Authors
- Sakaluk, J., Williams, A., & Biernat, M.
- Affiliations
- University of Kansas (US)
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2014-11
- Volume
- 9
- Issue
- 6
- Pages
- 652-660
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2133343019
- Cited by
- 32
- References
- 37
- Citations by year
- 2025: 1 · 2024: 2 · 2022: 3 · 2021: 3 · 2020: 2 · 2019: 4 · 2018: 4 · 2017: 4 · 2016: 4 · 2015: 4 · 2014: 1
- FWCI
- 12.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Perspectives on Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Topics
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- statistical reporting, data transparency, peer review, scientific integrity, reproducibility, publication process
- MeSH terms
- Humans, Psychology, Statistics as Topic, Scientific Misconduct, Peer Review, Research, Editorial Policies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.