Are Psychology Journals Anti-replication? A Snapshot of Editorial Practices
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Recent research in psychology has highlighted a number of replication problems in the discipline, with publication bias - the preference for publishing original and positive results, and a resistance to publishing negative results and replications- identified as one reason for replication failure. However, little empirical research exists to demonstrate that journals explicitly refuse to publish replications. We reviewed the instructions to authors and the published aims of 1151 psychology journals and examined whether they indicated that replications were permitted and accepted. We also examined whether journal practices differed across branches of the discipline, and whether editorial practices differed between low and high impact journals. Thirty three journals (3%) stated in their aims or instructions to authors that they accepted replications. There was no difference between high and low impact journals. The implications of these findings for psychology are discussed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 3×
“However, we would suggest that there are reasonably straightforward solutions to this problem and a number of specific solutions were suggested by Begley and Ellis (2012) in their discussion of replication failure in preclinical cancer trials.”
BackgroundBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 2×
“…for example, reporting the first of the Many Labs projects hosted by the Open Science Foundation, found a reasonably good rate of replication attempts: Of 13 replication attempts of classic and contemporary findings in social and cognitive psychology using 36 samples comprising 6344 participants,…”
BackgroundPublication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542 · cited 2×
“, 2011) and null findings are less likely to be submitted or accepted for publication as a result (Rosenthal, 1979; Franco et al., 2014).”
BackgroundContradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321 · cited 2×
“One ostensible reason for the current turmoil in the discipline has been psychology’s publication bias (Pashler and Wagenmakers, 2012).”
BackgroundNosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248 · cited 2×
“They should minimize the effect of ‘moderator variables,’ variables which may have been present in the replication that were not present in the original report and which are often cited by the authors of studies whose findings were not replicated as the reason for the failure to replicate.”
BackgroundInvestigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
Research works printed but not linked: 21
- American Psychological Society, 2015, Replication in psychological science., Psychol. Sci., 26, 1827, 10.1177/0956797615616374 · DOI · Paper page
- Bozarth, 1972, Signifying significant significance., Am. Psychol., 27, 774, 10.1037/h0038034 · DOI · Paper page
- Brown, 1976, Building a replication tradition in marketing., Marketing, 1776, 622
- Coursol, 1986, Effect of positive findings on submission and acceptance rates: a note on meta-analysis bias., Prof. Psychol. Res. Pract., 17, 136, 10.1037/0735-7028.17.2.136 · doi:10.1037/0735-7028.17.2.136 · Paper page
- Evanschitzky, 2010, Replications of forecasting research., Int. J. Forecast., 26, 4, 10.1016/j.ijforecast.2009.09.003 · DOI
- Evanschitzky, 2007, Replication research’s disturbing trend., J. Bus. Res., 60, 411, 10.1016/j.jbusres.2006.12.003 · DOI · Paper page
- Hubbard, 1994, Replications and extensions in marketing: rarely published but quite contrary., Int. J. Res. Mark., 11, 233, 10.1016/0167-8116(94)90003-5 · DOI · Paper page
- Hubbard, 1991, Replications in the finance literature: an empirical study., Q. J. Bus. Econ., 30, 70
- Hubbard, 1996, An empirical comparison of published replication research in accounting, economics, finance, management, and marketing., J. Bus. Res., 35, 153, 10.1016/0148-2963(95)00084-4 · DOI · Paper page
- Jones, 2010, An investigation of the prevalence of replication research in human factors., Hum. Factors, 52, 586, 10.1177/0018720810384394 · DOI · Paper page
- Kerr, 1977, Manuscript characteristics which influence acceptance for management and social science journals., Acad. Manag. J., 20, 132, 10.2307/255467 · DOI · Paper page
- Laws, 2013, Negativland-a home for all findings in psychology., BMC Psychol., 1, 10.1186/2050-7283-1-2 · DOI · Paper page
- Maxwell, 2015, Is psychology suffering from a replication crisis?, Am. Psychol., 70, 487, 10.1037/a0039400 · DOI · Paper page
- Neuliep, 1990, Editorial bias against replication research., J. Soc. Behav. Pers., 5, 85
- Neuliep, 1993a, Reviewer bias against replication research., J. Soc. Behav. Pers., 8, 21
- Neuliep, 1993b, Everyone was wrong: there are lots of replications out there., J. Soc. Behav. Pers., 8, 1
- Pashler, 2012, Is the replicability crisis overblown? Three arguments examined., Perspect. Psychol. Sci., 7, 531, 10.1177/1745691612463401 · DOI · Paper page
- Reid, 1981, Replication in advertising research: 977,1978, 1979., J. Advert., 10, 3, 10.1080/00913367.1981.10672750 · DOI · Paper page
- Rosenthal, 1979, The file drawer problem and tolerance for null results., Psychol. Bull., 86, 638, 10.1037/0033-2909.86.3.638 · DOI · Paper page
- Schooler, 2011, Unpublished results hide the decline effect., Nature, 470, 10.1038/470437a · DOI · Paper page
- Smart, 1964, The importance of negative results in psychological research., Can. Psychol., 5a, 225, 10.1037/h0083036 · DOI
Cited by
Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories
Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019) · Sage Open · doi:10.1177/2158244019829575 · Cited by 1,257
“Many journals appear to be reluctant to publish replications because they would have a negative influence on the citation rate, the impact factor, of the journal (G. N. Martin & Clarke, 2017).”
BackgroundZwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x17001972 · Cited by 444
Marsden, E., Morgan‐Short, K., Thompson, S., & Abugaber, D. (2018) · Language Learning · doi:10.1111/lang.12286 · Cited by 268
The Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137
“Many journals prevent or resist publication of null-fi ndings and replications 218 often due to perceived ‘ lack of contribution ’ , or prioritising novelty over incremental developments 219 – 221 .”
BackgroundSystematic review and meta-analyses of studies analysing instructions to authors from 1987 to 2017.
Malički, M., Jerončić, A., Aalbersberg, I. J., Bouter, L., & ter Riet, G. (2021) · Nature Communications · doi:10.1038/s41467-021-26027-y · Cited by 53
Metadata
- Authors
- Martin, G. N., & Clarke, R. M.
- Venue
- Frontiers in Psychology
- Publisher
- Frontiers Media SA
- Published
- 2017-04-11
- Volume
- 8
- Pages
- 523-523
- ISSN
- 1664-1078 (ISSN-L)
- OpenAlex
- W2607067892
- Cited by
- 116
- References
- 17
- FWCI
- 17.66
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.