The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Analytic Review as a Solution to the Misreporting of Statistical Results in Psychological Science
Published version of a preprint
Abstract
In this article, we propose analytic review (AR) as a solution to the problem of misreporting statistical results in psychological science. AR requires authors submitting manuscripts for publication to also submit the data file and syntax used during analyses. Regular reviewers or statistical experts then review reported analyses in order to verify that the analyses reported were actually conducted and that the statistical values are accurately reported. We begin by describing the problem of misreporting in psychology and introduce the basic AR process. We then highlight both primary and secondary benefits of adopting AR and describe different permutations of the AR system, each of which has its own strengths and limitations. We conclude by attempting to dispel three anticipated concerns about AR: that it will increase the workload placed on scholars, that it will infringe on the traditional peer-review process, and that it will hurt the image of the discipline of psychology. Although implementing AR will add one more step to the bureaucratic publication process, we believe it can be implemented in an efficient manner that would greatly assist in decreasing the frequency and impact of misreporting while also providing secondary benefits in other domains of scientific integrity.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Some journals, such as the American Journal of Political Science, have already gone a step further, and employ a third-party organization to check the analytic reproducibility of all articles before publishing them [47].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [47]
Citations
Cites
2 retracted
RETRACTED: Syncope in Adults With Pulmonary Arterial Hypertension
Le, R. J., Fenstad, E. R., Kremers, H. M., McCully, R. B., Frantz, R. P., McGoon, M. D., & Kane, G. C. (2011) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2011.04.026 · Cited by 30
Retracted(2012) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2012.04.006 · Cited by 1
RetractedThe file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
(1997) · Obesity Surgery · doi:10.1381/096089297765555359 · Cited by 3,916
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Publication Manual of the American Psychological Association
DeCleene, K. E., & Fogo, J. (2012) · Occupational Therapy In Health Care · doi:10.3109/07380577.2011.629024 · Cited by 3,167
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · cited 1×
“Investigations into the frequency of misreporting statistical values have yielded varying estimates of the prevalence of misreporting (e.g., 1.0%–2.0% in Fanelli, 2009; 22.0%–23.3% in John, Loewenstein, & Prelec, 2012).”
BackgroundInfluentialP-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
Research works printed but not linked: 2
- Herndon T., 2013, Cambridge Journal of Economics, 38, 1
- Simonsohn U., Journal of Experimental Psychology: General
Cited by
Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917747646 · Cited by 784
The prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510
“This solution would be in line with the notion of Analytic Review (Sakaluk, Williams, & Biernat, 2014), in which a reviewer receives the data file and syntax of a manuscript to check if the reported analyses were actually conducted and reported correctly.”
InfluentialHardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Henry Tessler, M., et al. (2018) · Royal Society Open Science · doi:10.1098/rsos.180448 · Cited by 336
Scientific Utopia III: Crowdsourcing Science
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850561 · Cited by 174
Hardwicke, T. E., Bohn, M., MacDonald, K., Hembacher, E., Nuijten, M. B., Peloquin, B. N., deMayo, B. E., Long, B., Yoon, E. J., & Frank, M. C. (2021) · Royal Society Open Science · doi:10.1098/rsos.201494 · Cited by 108
“Journals could also conduct independent assessment of analytic reproducibility prior to publication, as has been adopted by the American Journal of Political Science [22]; however, this would naturally require additional resources (for discussion see [23]).”
Method
Metadata
- Authors
- Sakaluk, J., Williams, A., & Biernat, M.
- Affiliations
- University of Kansas (US)
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2014-11
- Volume
- 9
- Issue
- 6
- Pages
- 652-660
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2133343019
- Cited by
- 32
- References
- 37
- Citations by year
- 2025: 1 · 2024: 2 · 2022: 3 · 2021: 3 · 2020: 2 · 2019: 4 · 2018: 4 · 2017: 4 · 2016: 4 · 2015: 4 · 2014: 1
- FWCI
- 12.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Perspectives on Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Topics
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- statistical reporting, data transparency, peer review, scientific integrity, reproducibility, publication process
- MeSH terms
- Humans, Psychology, Statistics as Topic, Scientific Misconduct, Peer Review, Research, Editorial Policies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Analytic Review as a Solution to the Misreporting of Statistical Results in Psychological Science. https://app.cochat.ai/papers/10.1177/1745691614549257
- The paper
- Sakaluk, J., Williams, A., & Biernat, M. (2014). Analytic Review as a Solution to the Misreporting of Statistical Results in Psychological Science. Perspectives on Psychological Science. https://doi.org/10.1177/1745691614549257
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.