CoChat
Sign Up
Cited by 1Scientific Computing and Data ManagementComputational and Text Analysis MethodsSoftware Engineering Research

Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches

Open access
Shah, Syed Mehtab Hussain, Hopfgartner, Frank, Bleier, Arnim·2026·Companion Proceedings of the ACM Web Conference 2026
DOI 10.1145/3774905.3795485

Published version of a preprint

Open this paper in CoChat — free account

Abstract

Reproducing computational research is often treated as a straightforward matter of re-running shared code on shared data. In practice, published analyses frequently fail even when materials are available, due to missing dependencies, brittle paths, version mismatches, and—more challenging—gaps in analytical logic. We investigate whether large language models (LLMs) and agentic AI systems can automate the routine but labor-intensive work of repairing such failures, thereby lowering the practical barrier to computational verification.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3774905.3795485

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In social science, prior works benchmark reproducibility while granting agents access to the original code.

    “In social science, prior works-Hu et al. (2025), Zhang et al. (2026), Shah et al. (2026), and Xu & Yang (2026)-benchmark reproducibility while granting agents access to the original code.”

    From Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results (2026) · cites this paper as Shah et al. (2026)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 31

  1. [1][1] Monya Baker. 2016. 1,500 scientists lift the lid on reproducibility. Nature 533, 7604 (2016), 452-454. doi:10.1038/533452a - · DOI · Paper page
  2. [3][3] Carl Boettiger and Dirk Eddelbuettel. 2017. An Introduction to Rocker: Docker Containers for R. arXiv:1710.03675 [cs.SE] https://arxiv.org/abs/1710.03675 - · arXiv
  3. [2][2] Lorena A. Barba. 2018. Terminologies for Reproducible Research. arXiv preprint arXiv:1802.03311 (2018). https://arxiv.org/abs/1802.03311 - · arXiv
  4. [4][4] Nate Breznau, Eike Mark Rinke, Alexander Wuttke, Muna Adem, and others. 2025. The reliability of replications: a study in computational reproductions. Royal Society Open Science 12, 3 (2025), 241038. doi:10.1098/rsos.241038 - · DOI
  5. [6][6] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs.LG] https://arxiv.org/abs/2107.03374 - · arXiv
  6. [5][5] Chung-hong Chan, Tim Schatto-Eckrodt, and Johannes Gruber. 2024. What Makes Computational Communication Science (Ir)Reproducible? Computational Communication Research (2024). doi:10.5117/CCR2024.1.5.CHAN - · DOI
  7. [7][7] Fernando Chirigati, Rémi Rampin, Dennis Shasha, and Juliana Freire. 2016. ReproZip: Computational Reproducibility With Ease. In Proceedings of the 2016 International Conference on Management of Data (San Francisco, California, USA) (SIGMOD'16) . Association for Computing Machinery, New York, NY, USA, 2085-2088. doi:10.1145/2882903.2899401 - · DOI · Paper page
  8. [9][9] Lázaro Costa, Susana Barbosa, and Jácome Cunha. 2025. A Dataset For Computational Reproducibility. arXiv:2504.08684 [cs.SE] https://arxiv.org/abs/2504.08684 - · arXiv
  9. [8][8] Jon F. Claerbout and Martin Karrenbach. 1992. Electronic documents give reproducible research a new meaning. In SEG Technical Program Expanded Abstracts . 601-604. doi:10.1190/1.1822162 - · DOI
  10. [10][10] Andor Diera, Abdelhalim Dahou, Lukas Galke, Fabian Karl, Florian Sihler, and Ansgar Scherp. 2023. GenCodeSearchNet: A Benchmark Test Suite for Evaluating Generalization in Programming Language Understanding. arXiv:2311.09707 [cs.CL] https://arxiv.org/abs/2311.09707 - · arXiv
  11. [11][11] Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Bodapati, Aram Galstyan, Azton Wells, Roy Schwartz, Eliu A Huerta, and Hao Peng. 2025. Context Length Alone Hurts LLM Performance Despite Perfect Retrieval. arXiv:2510.05381 [cs.CL] https://arxiv.org/abs/2510.05381 - · arXiv
  12. [13][13] TomE. Hardwicke, Maya B. Mathur, Kyle MacDonald, Gustav Nilsonne, George C. Banks, Mallory C. Kidwell, Alicia Hofelich Mohr, Elizabeth Clayton, Erica J. Yoon, Michael Henry Tessler, Richie L. Lenne, Sara Altman, Bria Long, and Michael C. Frank. 2018. Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition. Royal Society Open Science 5, 8 (08 2018), 180448. https://doi.org/10.1098/rsos.180448 - · DOI · Paper page
  13. [15][15] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Trans. Inf. Syst. 43, 2, Article 42 (Jan. 2025), 55 pages. doi:10.1145/3703155 - · DOI · Paper page
  14. [14][14] Chuxuan Hu, Liyun Zhang, Yeji Lim, Aum Wadhwani, Austin Peters, and Daniel Kang. 2025. REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research? arXiv:2507.18901 [cs.CL] https://arxiv.org/abs/2507. 18901 - · arXiv
  15. [16][16] S. Kambouris, D. P. Wilkinson, E. T. Smith, and F. Fidler. 2024. Computationally reproducing results from meta-analyses in ecology and evolutionary biology 17. using shared code and data. PLOS ONE 19, 3 (2024), e0300333. doi:10.1371/journal. pone.0300333 - · DOI
  16. [18][18] National Academies of Sciences, Medicine, Policy, Global Affairs, Board on Research Data, Information, Division on Engineering, Physical Sciences, Committee on Applied, Theoretical Statistics, et al. 2019. Reproducibility and replicability in science . National Academies Press. doi:10.17226/25303 - · DOI · Paper page
  17. [17][17] David M. Liu and Matthew J. Salganik. 2019. Successes and Struggles with Computational Reproducibility: Lessons from the Fragile Families Challenge. Socius : sociological research for a dynamic world 5 (2019). https://api.semanticscholar. org/CorpusID:203527468 -
  18. [21][21] Tim Reason, Emma Benbow, Julia Langham, Andy Gimblett, Sven L. Klijn, and Bill Malcolm. 2024. Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models. PharmacoEconomics - Open 8, 2 (2024), 205-220. doi:10.1007/s41669-024-00476-9 - · DOI · Paper page
  19. [22][22] Ahmadreza Saboor Yaraghi, Darren Holden, Nafiseh Kahani, and Lionel Briand. 2025. Automated Test Case Repair Using Language Models. IEEE Transactions on Software Engineering 51, 4 (April 2025), 1104-1133. doi:10.1109/tse.2025.3541166 - · DOI
  20. [24][24] David Schoch, Chung-hong Chan, Claudia Wagner, and Arnim Bleier. 2024. Computational reproducibility in computational social science. EPJ Data Science 13, 1 (2024), 75. doi:10.1140/epjds/s13688-024-00514-w - · DOI
  21. [23][23] Lorraine Saju, Tobias Holtdirk, Meetkumar Pravinbhai Mangroliya, and Arnim Bleier. 2025. Computational Reproducibility of R Code Supplements on OSF. ICWSM. doi:10.36190/2025.49 - · DOI
  22. [25][25] Zachary S. Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan. 2024. CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark. arXiv:2409.11363 [cs.CL] https://arxiv.org/abs/2409.11363 - · arXiv
  23. [27][27] Jiajun Sun, Fengjie Li, Xinzhu Qi, Hongyu Zhang, and Jiajun Jiang. 2025. Empirical Evaluation of Large Language Models in Automated Program Repair. arXiv:2506.13186 [cs.SE] https://arxiv.org/abs/2506.13186 - · arXiv
  24. [26][26] Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. 2025. PaperBench: Evaluating AI's Ability to Replicate AI Research. arXiv:2504.01848 [cs.AI] https://arxiv.org/ abs/2504.01848 - · arXiv
  25. [28][28] Hao Tang, Keya Hu, Jin Peng Zhou, Sicheng Zhong, Wei-Long Zheng, Xujie Si, and Kevin Ellis. 2024. Code Repair with LLMs gives an Exploration-Exploitation Tradeoff. arXiv:2405.17503 [cs.SE] https://arxiv.org/abs/2405.17503 - · arXiv
  26. [30][30] Magdalena Wysocka, Oskar Wysocki, Maxime Delmas, Vincent Mutel, and André Freitas. 2024. Large Language Models, scientific knowledge and factuality: A framework to streamline human expert evaluation. Journal of Biomedical Informatics 158 (2024), 104724. doi:10.1016/j.jbi.2024.104724 - · DOI
  27. [29][29] Ana Trisovic, Matthew K. Lau, Thomas Pasquier, and Mercè Crosas. 2022. A large-scale study on research code quality and execution. Scientific Data 9, 1 (2022), 60. doi:10.1038/s41597-022-01143-6 - · DOI
  28. [31][31] Weiqing Yang, Hanbin Wang, Zhenghao Liu, Xinze Li, Yukun Yan, Shuo Wang, Yu Gu, Minghe Yu, Zhiyuan Liu, and Ge Yu. 2025. COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis. arXiv:2408.05006 [cs.SE] https://arxiv.org/abs/2408.05006 - · arXiv
  29. [33][33] Linhao Zhang, Tong Xia, Jinghua Piao, Lizhen Cui, and Yong Li. 2026. PaperRepro: Automated Computational Reproducibility Assessment for Social Science Papers. arXiv:2603.00058 [cs.CY] https://arxiv.org/abs/2603.00058 - · arXiv
  30. [32][32] Alex L. Zhang, Tim Kraska, and Omar Khattab. 2025. Recursive Language Models. arXiv:2512.24601 [cs.AI] https://arxiv.org/abs/2512.24601 - · arXiv
  31. [34][34] Yanbo Zhang, Sumeer A. Khan, Adnan Mahmud, Huck Yang, Alexander Lavin, Michael Levin, Jeremy Frey, Jared Dunnmon, James Evans, Alan Bundy, Saso Dzeroski, Jesper Tegner, and Hector Zenil. 2025. Exploring the role of large language models in the scientific method: from hypothesis to discovery. npj Artificial Intelligence 1, 1 (2025), 14. doi:10.1038/s44387-025-00019-5 · DOI

Cited by

Sorted by citations

Authors
Shah, S. M. H., Hopfgartner, F., & Bleier, A.
Affiliations
  • GESIS - Leibniz-Institut für Sozialwissenschaften, Cologne, Germany
  • Universität Koblenz (DE)
  • GESIS - Leibniz Institute for the Social Sciences (DE)
ORCID
Venue
Companion Proceedings of the ACM Web Conference 2026
Publisher
ACM
Published
2026-05-28
Pages
989-998
DOI
10.1145/3774905.3795485
OpenAlex
W7155606551
Cited by
0
References
0
FWCI
0.00
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • International World Wide Web Conference (WWW) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
computational reproducibility, large language models, agentic AI, agent-based approach, computational verification, social science research, automated program repair

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review