Investigating the reproducibility of the social and behavioural sciences
Open accessAbstract
Published claims should be reproducible, yielding the same result when applying the same analysis to the same data. We assessed reproducibility in a stratified random sample of 600 papers published from 2009 to 2018 in 62 journals spanning the social and behavioural sciences. Authors of 144 (24.0% [95% CI 20.8 - 27.6%]) papers made data available to assess reproducibility, and for 38 others, we obtained source data to reconstruct the dataset. We assessed 143 of the 182 available datasets and found that 76.6 (53.6% [95% CI 45.8 - 60.7%]) papers were rated as precisely reproducible and 105.0 (73.5% [95% CI 66.4 - 80.0%]) papers as at least approximately reproducible (within 15% of the original effects or within .05 of original p-values) after inverse weighting each of the 551 claims by the number of claims per paper. We observed higher reproducibility for papers from Political Science and Economics than other fields, for more recent than older papers, and for papers from journals that required data sharing. Implementing measures to verify that research is reproducible is needed to support trustworthiness in the complex enterprise of knowledge production.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Such crowd-sourced reproduction work requires considerable resources-hundreds of trained researchers to collectively assess only a negligible fraction of the published literature.
“Yet such work requires considerable resources-hundreds of trained researchers to collectively assess only a negligible fraction of the published literature (Brodeur et al., 2026; Miske et al., 2026; Tyner et al., 2026).”
From Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results (2026) · cites this paper as Miske et al., 2026
The Abstract reports that the project examined 600 papers across 62 journals but assessed reproducibility for only 143 datasets, supporting the point that a large collaborative effort covered a limited set of studies. The author list shows a large collaboration, but the passages do not establish that hundreds of trained researchers performed the assessments or quantify the resources required. They also do not measure the assessed papers as a fraction of all published literature. (LLM, conf=0.82)
Experimental — accuracy not yet measuredRecent crowd-sourced reproduction efforts have revealed substantial variation in reproducibility across social science disciplines.
“Recent crowd-sourced reproduction efforts have revealed substantial variation in reproducibility across social science disciplines.”
From Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results (2026) · cites this paper as Miske et al., 2026
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cited by
Sorted by citations
Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results
Kohler, B., Zollikofer, D., Einsiedler, J., Hoyle, A., & Ash, E. (2026) · arXiv · doi:10.48550/arxiv.2604.21965 · Cited by 1 · see citing works
- Authors
- Miske, O., Abatayo, A. L., Daley, M., Dirzo, M., Fox, N. W., Haber, N., Hahn, K., Kline Struhl, M., Mawhinney, B., Silverstein, P., Stankov, T., Tyner, A. H., Adamkovic, M., Alzahawi, S., Anafinova, S., Awtrey, E. C., Axxe, E., Bailey, J., Bakker, B. N., Balaji, A., Baník, G., Bartoš, F., berkman, H., Berry, Z., Bethke, F., Brady, T. F., Breznau, N., Capitan, S., Capitán, T., Caquelin, L., Cheng, K. J. G., Chopik, W. J., Clochard, G.-J., Coupe, T., Cummins, J., Burak, E. G. D., Duan, J., Esterling, K., Evans, T. R., Fiala, N., Field, J., Gay, V., Geng, J., Gereke, J., Gleibs, I. H., Gourdon-Kanhukamwe, A., Grigoryev, D., Gunby, N., Hanel, P. H. P., Hong, S., Houlihan, S. D., Huntington-Klein, N., Izydorczak, K., Jankowsky, K., Jonas, K., Kačmár, P., Kapoor, H., Karcher, S., Kołczyńska, M., Kretschmer, D., Lazarevic, L., Leahy, K. E., Lee, J. C., Limnios, A. C., Liu, A.-C., Lloyd, J. W., Lopez-Nicolas, R., Lou, N. M., Lucas, R., Maier, M., Mallinson, D. J., Martončik, M., McCall, M. C., Mehta, N. P., Méndez-Chacón, E., Michalak, J., Molden, D. C., Mushtaq, F., Neuendorf, C., Nichols, A. L., Nilsonne, G., O'Boyle, E., Oh, J., Ostermann, T., OYEBANJO, A., Panczak, R., Pavlov, Y. G., Pavlović, Z., Peter, N., Peters, K. O., Porter, N. D., Purol, M., Puthillam, A., Ramljak, M., Reader, A. T., Reed, W. R., Röer, J. P., Ropovik, I., Savi, A. O., Schmidt, K., Schnabel, L., Sevigny, E. L., Shaki, S., Shakya, S., Soh, A., Somo, A., Sonmez, F., Strømland, E. A., Suchow, J., Szabelska, A., Tagat, A., Tutor, M., Urbanska, K., Van Dessel, P., Vargo, E. J., Vo, D. T. H., Volkman, V., Wang, K., Wichman, A. L., Williams, J. R., Winter, F., Wintermantel, F., Zhang, N., Ziano, I., Zogmaister, C., Zupan, Z., Nosek, B. A., & Errington, T. M.
- Affiliations
- Arizona State University (US)
- Center for Open Science (US)
- Center For Open Science
- University of California, Riverside (US)
- Virginia Tech (US)
- Massachusetts Institute of Technology (US)
- Universiteit Gent
- Ghent University (BE)
- ORCID
- Miske, Olivia
- Abatayo, Anna Lou
- Daley, Macie
- Dirzo, Mirka
- Fox, Nicholas William
- Haber, Noah
- Mawhinney, Brinna
- Alzahawi, Shilaan
- Anafinova, Saule
- Awtrey, Eli C.
- Axxe, Erick
- Bailey, James
- Bakker, Bert N.
- Bartoš, František
- berkman, henk
- Berry, Zachariah
- Bethke, Felix
- Brady, Timothy F.
- Breznau, Nate
- Capitán, Tabaré
- Caquelin, Laura
- Cheng, Kent Jason Go
- Chopik, William J.
- Burak, Elif Gizem Demirag
- Esterling, Kevin
- Evans, Thomas Rhys
- Field, James
- Gay, Victor
- Geng, Jing
- Gereke, Johanna
- Gleibs, Ilka Helene
- Gourdon-Kanhukamwe, Amélie
- Grigoryev, Dmitry
- Hanel, Paul H. P.
- Hong, Sanghyun
- Houlihan, Sean Dae
- Huntington-Klein, Nick
- Izydorczak, Kamil
- Jonas, Kai
- Kačmár, Pavol
- Kapoor, Hansika
- Karcher, Sebastian
- Kołczyńska, Marta
- Kretschmer, David
- Leahy, Katelin E.
- Lee, Jessica C.
- Limnios, Angelo Christopher
- Liu, An-Chiao
- Lloyd, John Wills
- Lopez-Nicolas, Ruben
- Lou, Nigel Mantou
- Maier, Maximilian
- Martončik, Marcel
- McCall, Michael C.
- Michalak, Johannes
- Molden, Daniel C.
- Neuendorf, Claudia
- Nichols, Austin Lee
- Nilsonne, Gustav
- Oh, Jeewon
- Ostermann, Thomas
- OYEBANJO, Abiola
- Panczak, Radoslaw
- Pavlov, Yuri G.
- Pavlović, Zoran
- Peters, Kim Olivia
- Porter, Nathaniel D
- Purol, Mariah
- Puthillam, Arathy
- Ramljak, Marco
- Reader, Arran T
- Reed, W. Robert
- Röer, Jan Philipp
- Savi, Alexander Olof
- Schnabel, Landon
- Shaki, Samuel
- Shakya, Shishir
- Somo, Angela
- Suchow, Jordan
- Szabelska, Anna
- Tagat, Anirudh
- Tutor, Melba
- Urbanska, Karolina
- Van Dessel, Pieter
- Vargo, Elisabeth Julie
- Vo, Diem Thi Hong
- Volkman, Victor
- Wichman, Aaron Lee
- Williams, Jamal Rodgers
- Winter, Fabian
- Zhang, Nan
- Ziano, Ignazio
- Zogmaister, Cristina
- Zupan, Zorana
- Nosek, Brian A.
- Errington, Timothy M.
- Publisher
- Center for Open Science
- Published
- 2026-04-02
- OpenAlex
- W7148331934
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- reproducibility, social and behavioral sciences, data availability, research trustworthiness, data sharing policy, journal requirements, reproducibility assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
