Lessons from a Decade of Replications at the Quarterly Journal of Political Science
Open accessAbstract
To allow researchers to investigate not only whether a paper’s methods are theoretically sound but also whether they have been properly implemented and are robust to alternative specifications, it is necessary that published papers be accompanied by their underlying data and code. This article describes experiences and lessons learned at the Quarterly Journal of Political Science since it began requiring authors to provide this type of replication code in 2005. It finds that of the 24 empirical papers subjected to in-house replication review since September 2012, only four packages did not require any modifications. Most troubling, 14 packages (58%) had results in the paper that differed from those generated by the author’s own code. Based on these experiences, this article presents a set of guidelines for authors and journals for improving the reliability and usability of replication packages.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Eubank conducted a pre-publication assessment of articles submitted to the Quarterly Journal of Political Science, and found that only four out of 24 articles (approx. 17%) were reproducible without author assistance.
“For example, Eubank [19] conducted a pre-publication assessment of articles submitted to the Quarterly Journal of Political Science, and found that only four out of 24 articles (approx. 17%) were reproducible without author assistance.”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal <i>Cognition</i> (2018) · cites this paper as [19]
The Abstract reports that only four of 24 empirical papers’ replication packages required no modifications, and the In-House Replication section confirms that the review took place before publication. It does not equate needing a modification with being unreproducible without author assistance, so the claim’s stated measure of reproducibility is not established. (LLM, conf=0.85)
Experimental — accuracy not yet measuredA smattering of studies have sought to investigate analytic reproducibility in several scientific disciplines, and most have encountered substantial reproducibility problems.
“A smattering of studies have sought to investigate this issue in several scientific disciplines, and most have encountered substantial reproducibility problems [19–21].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal <i>Cognition</i> (2018) · cites this paper as [19]
This is comparable to the success rates observed in other assessments of analytic reproducibility in political science [19] and economics [20], but is substantially worse than a recent study of clinical RCTs [22].
“This is comparable to the success rates observed in other assessments of analytic reproducibility in political science [19] and economics [20], but is substantially worse than a recent study of clinical RCTs [22].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal <i>Cognition</i> (2018) · cites this paper as [19]
Policies that mandate sharing of analysis scripts have already been implemented at some journals, for example in economics and political science.
“Policies that mandate sharing of analysis scripts have already been implemented at some journals, for example in economics and political science [19,45,46].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal <i>Cognition</i> (2018) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Still being checked.
Cited by
Sorted by citations
Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Henry Tessler, M., et al. (2018) · Royal Society Open Science · doi:10.1098/rsos.180448 · Cited by 336 · see citing works
A Practical Guide for Transparency in Psychological Science
Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., IJzerman, H., Nilsonne, G., Vanpaemel, W., & Frank, M. C. (2018) · Collabra Psychology · doi:10.1525/collabra.158 · Cited by 233 · see citing works
Hardwicke, T. E., Wallach, J. D., Kidwell, M. C., Bendixen, T., Crüwell, S., & Ioannidis, J. P. A. (2020) · Royal Society Open Science · doi:10.1098/rsos.190806 · Cited by 217 · see citing works
Hardwicke, T. E., Bohn, M., MacDonald, K., Hembacher, E., Nuijten, M. B., Peloquin, B. N., deMayo, B. E., Long, B., Yoon, E. J., & Frank, M. C. (2021) · Royal Society Open Science · doi:10.1098/rsos.201494 · Cited by 108 · see citing works
Computational Reproducibility via Containers in Psychology
Clyburne-Sherin, A., Fei, X., & Green, S. A. (2019) · Meta-Psychology · doi:10.15626/mp.2018.892 · Cited by 69 · see citing works
Artner, R., Verliefde, T., Steegen, S., Gomes, S. L. B. T., Traets, F., Tuerlinckx, F., & Vanpaemel, W. (2020) · Psychological Methods · doi:10.1037/met0000365 · Cited by 63 · see citing works
Laurinavichyute, A., Yadav, H., & Vasishth, S. (2022) · Journal of Memory and Language · doi:10.1016/j.jml.2022.104332 · Cited by 60 · see citing works
Reproducibility in the Social Sciences
Moody, J. W., Keister, L. A., & Ramos, M. C. (2022) · Annual Review of Sociology · doi:10.1146/annurev-soc-090221-035954 · Cited by 55 · see citing works
Hardwicke, T. E., Mathur, M. B., MacDonald, K. E., Nilsonne, G., Banks, G. C., Kidwell, M. C., Mohr, A. H., Clayton, E., Yoon, E. J., Tessler, M. H., et al. (2018) · doi:10.31222/osf.io/39cfb · Cited by 60 · see citing works
Reproducibility in Management Science
Fišar, M., Greiner, B., Huber, C., Katok, E., & Özkes, A. (2023) · Management Science · doi:10.1287/mnsc.2023.03556 · Cited by 51 · see citing works
A practical guide for transparency in psychological sciencePreprint
Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., IJzerman, H., Nilsonne, G., Vanpaemel, W., & Frank, M. C. (2018) · doi:10.31234/osf.io/rtygm · Cited by 46 · see citing works
Investigating the reproducibility of the social and behavioural sciences
Miske, O., Abatayo, A. L., Daley, M., Dirzo, M., Fox, N. W., Haber, N., Hahn, K. M., Struhl, M. K., Mawhinney, B., Silverstein, P., et al. (2026) · Nature · doi:10.1038/s41586-026-10203-5 · Cited by 28 · see citing works
Computational Reproducibility via Containers in Social PsychologyPreprint
Clyburne-Sherin, A., Fei, X., & Green, S. A. (2018) · doi:10.31234/osf.io/mf82t · Cited by 19 · see citing works
Balafoutas, L., Celse, J., Karakostas, A., & Umashev, N. (2024) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2024.102327 · Cited by 18 · see citing works
Preregistration and Registered Reports in Sociology: Strengths, Weaknesses, and Other Considerations
Manago, B. N. (2023) · The American Sociologist · doi:10.1007/s12108-023-09563-6 · Cited by 13 · see citing works
The Rockerverse: Packages and Applications for Containerisation with R
Nüst, D., Eddelbüttel, D., Bennett, D. S., Cannoodt, R., Clark, D., Daróczi, G., Edmondson, M., Fay, C., Hughes, E., Kjeldgaard, L., et al. (2020) · The R Journal · doi:10.32614/rj-2020-007 · Cited by 13 · see citing works
Forschungstransparenz als hohes wissenschaftliches Gut stärken
Schönbrodt, F. D., Maier, M., Heene, M., & Bühner, M. (2018) · Psychologische Rundschau · doi:10.1026/0033-3042/a000386 · Cited by 11 · see citing works
The Reliability of Replications: A Study in Computational ReproductionsPreprint
Breznau, N., Rinke, E. M., Wuttke, A., Nguyen, H. H. V., Adem, M., Adriaans, J., Akdenız, E., Álvarez-Benjumea, A., Andersen, H. K., Auer, D., et al. (2021) · doi:10.31235/osf.io/j7qta · Cited by 8 · see citing works
I Saw You in the Crowd: Credibility, Reproducibility, and Meta-Utility
Breznau, N. (2021) · PS Political Science & Politics · doi:10.1017/s1049096520000980 · Cited by 8 · see citing works
Reproducibility in Management SciencePreprint
Fišar, M., Greiner, B., Huber, C., Katok, E., Özkes, A., & Management Science Reproducibility Collaboration (2023) · doi:10.31219/osf.io/mydzv · Cited by 9 · see citing works
Heather, A., Monks, T., Harper, A. L., Mustafee, N., & Mayne, A. (2025) · Journal of Simulation · doi:10.1080/17477778.2025.2552177 · Cited by 7 · see citing works
Teaching Constructive Replications in the Social SciencesPreprint
Bauer, G., Breznau, N., Gereke, J., Höffler, J. H., Janz, N., Rahal, R.-M., Rennstich, J. K., & Soiné, H. (2022) · doi:10.31222/osf.io/ejkws · Cited by 7 · see citing works
The reliability of replications: a study in computational reproductions
Breznau, N., Rinke, E. M., Wuttke, A., Adem, M., Adriaans, J., Akdeniz, E., Álvarez-Benjumea, A., Andersen, H. K., Auer, D., Azevedo, F., et al. (2025) · Royal Society Open Science · doi:10.1098/rsos.241038 · Cited by 6 · see citing works
Teaching Constructive Replications in the Behavioral and Social Sciences Using Quantitative Data
Bauer, G., Breznau, N., Gereke, J., Höffler, J. H., Janz, N., Rahal, R.-M., Rennstich, J. K., & Soiné, H. (2023) · Teaching of Psychology · doi:10.1177/00986283231219503 · Cited by 12 · see citing works
Embedding Data Skills in Research Methods Education: Preparing Students for Reproducible ResearchPreprint
McAleer, P., Stack, N., Woods, H. C., DeBruine, L. M., Paterson, H., Nordmann, E., Kuepper-Tetzel, C. E., & Barr, D. J. (2022) · doi:10.31234/osf.io/hq68s · Cited by 6 · see citing works
Fink, L., & Marcus, J. (2024) · Economic Inquiry · doi:10.1111/ecin.13267 · Cited by 5 · see citing works
Tsui, A. S. M., Marchman, V. A., & Frank, M. C. (2021) · Advances in child development and behavior · doi:10.1016/bs.acdb.2021.04.003 · Cited by 5 · see citing works
Reproducibility, replicability, robustness, and generalizability in corpus linguistics
Flanagan, J. (2025) · International Journal of Corpus Linguistics · doi:10.1075/ijcl.24113.fla · Cited by 3 · see citing works
Alvarez, R. M., & Heuberger, S. (2021) · PS Political Science & Politics · doi:10.1017/s1049096521001062 · Cited by 4 · see citing works
Breznau, N., Rinke, E. M., Wuttke, A., Adem, M., Adriaans, J., Álvarez-Benjumea, A., Andersen, H. K., Auer, D., Azevedo, F., Bahnsen, O., et al. (2019) · doi:10.31235/osf.io/6j9qb · Cited by 4 · see citing works
Prosser, A. M. B., Brown, O., Augustine, G. L., & Ellis, D. A. (2024) · doi:10.31235/osf.io/ntf73 · Cited by 4 · see citing works
Reproducibility in Management SciencePreprint
Fišar, M., Greiner, B., Huber, C., Katok, E., & Özkes, A. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4620006 · Cited by 1 · see citing works
LabFam Individual Biographies harmonised family and employment histories based on panel surveysPreprint
Weychert, E., Osiewalska, B., van der Velde, L., & Matysiak, A. (2025) · Research Square · doi:10.21203/rs.3.rs-8376548/v1 · Cited by 1 · see citing works
Jamshidi‐Naeini, Y., Vorland, C. J., Kapoor, P., Ortyl, B. K., Mineo, J., Still, L., Sorsor, K., Rodney, S., Tooze, X. A., Flickinger, B., et al. (2026) · Nutrition and Diabetes · doi:10.1038/s41387-026-00444-8 · Cited by 1 · see citing works
Heather, A., Monks, T., Harper, A. L., Alidoost, F., Challen, R., Slater, T., & Mustafee, N. (2026) · NIHR Open Research · doi:10.3310/nihropenres.14296.1
- Authors
- Eubank, N.
- Affiliations
- Stanford University (US)
- Corresponding author
- Eubank, Nicholas
- ORCID
- Eubank, Nicholas
- Venue
- Political Science and Politics
- Publisher
- Cambridge University Press (CUP)
- Published
- 2016-04
- Volume
- 49
- Issue
- 2
- Pages
- 273-276
- ISSN
- 0030-8269 (ISSN-L)
- 1049-0965
- 1537-5935
- OpenAlex
- W2341138123
- Cited by
- 35
- References
- 0
- Citations by year
- 2026: 3 · 2025: 4 · 2024: 3 · 2023: 5 · 2022: 4 · 2021: 5 · 2020: 3 · 2019: 2 · 2018: 6
- FWCI
- 13.49
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- PS Political Science & Politics (publishedVersion)
- Topics
- Social Sciences › Political Science and International Relations › Political Science Research and Education
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- replication, replication code, reproducibility, author guidelines, editorial policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
