CoChat
Sign Up
Cited by 19Software Engineering ResearchSoftware Engineering Techniques and PracticesScientific Computing and Data Management

Can GPT-4 Replicate Empirical Software Engineering Research?

Open access
Liang, Jenny T., Badea, Carmen, Bird, Christian, DeLine, Robert, Ford, Denae, Forsgren, Nicole, Zimmermann, Thomas·2024·Proceedings of the ACM on Software Engineering
DOI 10.1145/3660767
Open this paper in CoChat — free account

Abstract

Empirical software engineering research on production systems has brought forth a better understanding of the software engineering process for practitioners and researchers alike. However, only a small subset of production systems is studied, limiting the impact of this research. While software engineering practitioners could benefit from replicating research on their own data, this poses its own set of challenges, since performing replications requires a deep understanding of research methodologies and subtle nuances in software engineering data. Given that large language models (LLMs), such as GPT-4, show promise in tackling both software engineering- and science-related tasks, these models could help replicate and thus democratize empirical software engineering research. In this paper, we examine GPT-4’s abilities to perform replications of empirical software engineering research on new data. We specifically study their ability to surface assumptions made in empirical software engineering research methodologies, as well as their ability to plan and generate code for analysis pipelines on seven empirical software engineering papers. We perform a user study with 14 participants with software engineering research expertise, who evaluate GPT-4-generated assumptions and analysis plans (i.e., a list of module specifications) from the papers. We find that GPT-4 is able to surface correct assumptions, but struggles to generate ones that apply common knowledge about software engineering data. In a manual analysis of the generated code, we find that the GPT-4-generated code contains correct high-level logic, given a subset of the methodology. However, the code contains many small implementation-level errors, reflecting a lack of software engineering knowledge. Our findings have implications for leveraging LLMs for software engineering research as well as practitioner data scientists in software teams.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3660767

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLM-generated programs have correct high-level structures but implementation-level errors, such as missing steps or wrong API usage.

    “For most failed tasks, we share similar findings with Liang et al. (2024) that LLM-generated programs have correct high-level structures but implementation-level errors, such as missing steps or wrong API usage.”

    From ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery (2024) · cites this paper as Liang et al. (2024)

    The Conclusion (p. 20) says GPT-4-generated code was correct in high-level structure but could contain lower-level errors. Section 4.2 (pp. 16–17) documents missing methodology steps and hallucinated APIs. The claim broadens this GPT-4 finding to LLM-generated programs generally, while Threats to Validity (p. 19) says its generalizability to other LLMs is unclear. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 58

  1. [3][3] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. PaLM 2 Technical Report. arXiv preprint arXiv:2305.10403 (2023). - · arXiv
  2. [4][4] Sören Auer, Dante AC Barone, Cassiano Bartz, Eduardo G Cortes, Mohamad Yaser Jaradeh, Oliver Karras, Manolis Koubarakis, Dmitry Mouromtsev, Dmitrii Pliukhin, Daniil Radyush, et al. 2023. The SciQA Scientific Question Answering Benchmark for Scholarly Knowledge. Scientific Reports 13, 1 (2023), 7240. https://doi.org/10.1038/s41598023-33607-z - · DOI
  3. [5][5] Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. 2022. PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts. In Annual Meeting of the Association for Computational Linguistics (ACL): System Demonstrations . Association for Computational Linguistics, 93-104. https://doi.org/10.18653/v1/2022.acl-demo.9 - · DOI · Paper page
  4. [6][6] Sebastian Baltes and Stephan Diehl. 2018. Towards a Theory of Software Development Expertise. In ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) . 187-200. https://doi.org/10.1145/3236024.3236061 - · DOI
  5. [8][8] AndrewBegel and Thomas Zimmermann. 2014. Analyze This! 145 Questions for Data Scientists in Software Engineering. In IEEE/ACM International Conference on Software Engineering (ICSE) . 12-23. https://doi.org/10.1145/2568225.2568233 - · DOI
  6. [9][9] Christian Bird, Nachiappan Nagappan, Premkumar Devanbu, Harald Gall, and Brendan Murphy. 2009. Does Distributed Development Affect Software Quality? An Empirical Case Study of Windows Vista. Commun. ACM 52, 8 (2009), 85-93. https://doi.org/10.1109/ICSE.2009.5070550 - · DOI
  7. [10][10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models Are Few-shot Learners. Advances in Neural Information Processing Systems (NeurIPS) 33 (2020), 1877-1901. -
  8. [11][11] J. Carver, J. VanVoorhis, and V. Basili. 2004. Understanding the Impact of Assumptions on Experimental Validity. In International Symposium on Empirical Software Engineering (ISESE) . 251-260. https://doi.org/10.1109/ISESE.2004. 1334912 - · DOI
  9. [12][12] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021). - · arXiv
  10. [13][13] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. PaLM: Scaling Language Modeling with Pathways. arXiv preprint arXiv:2204.02311 (2022). - · arXiv
  11. [14][14] Andy Cockburn, Pierre Dragicevic, Lonni Besançon, and Carl Gutwin. 2020. Threats of a Replication Crisis in Empirical Computer Science. Commun. ACM 63, 8 (2020), 70-79. https://doi.org/10.1145/3360311 - · DOI
  12. [15][15] Robert A DeLine. 2021. Glinda: Supporting Data Science with Live Programming, GUIs and a Domain-specific Language. In ACM CHI Conference on Human Factors in Computing Systems . 1-11. https://doi.org/10.1145/3411764.3445267 - · DOI
  13. [17][17] Jon Eyolfson, Lin Tan, and Patrick Lam. 2011. Do Time of Day and Developer Experience Affect Commit Bugginess?. In Working Conference on Mining Software Repositories (MSR) . 153-162. https://doi.org/10.1145/1985441.1985464 - · DOI
  14. [18][18] Sakina Fatima, Taher A Ghaleb, and Lionel Briand. 2022. Flakify: A Black-box, Language Model-based Predictor for Flaky Tests. IEEE Transactions on Software Engineering (2022). https://doi.org/10.1109/TSE.2022.3201209 - · DOI
  15. [19][19] Beat Fluri, Michael Wursch, Martin PInzger, and Harald Gall. 2007. Change Distilling: Tree Differencing for Finegrained Source Code Change Extraction. IEEE Transactions on Software Engineering 33, 11 (2007), 725-743. https: //doi.org/10.1109/TSE.2007.70731 - · DOI
  16. [20][20] Bent Flyvbjerg. 2006. Five misunderstandings about case-study research. Qualitative Inquiry 12, 2 (2006), 219-245. https://doi.org/10.1177/1077800405284363 - · DOI · Paper page
  17. [21][21] Denae Ford, Margaret-Anne Storey, Thomas Zimmermann, Christian Bird, Sonia Jaffe, Chandra Maddila, Jenna L Butler, Brian Houck, and Nachiappan Nagappan. 2021. A Tale of Two Cities: Software Developers Working from Home during the Covid-19 Pandemic. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 2 (2021), 1-37. -
  18. [22][22] Enrico Fregnan, Larissa Braz, Marco D'Ambros, Gül Çalıklı, and Alberto Bacchelli. 2022. First Come First Served: The Impact of File Position on Code Review. In ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) . 483-494. https://doi.org/10.1145/3540250.3549177 - · DOI
  19. [23][23] Fan Gao, Hang Jiang, Moritz Blum, Jinghui Lu, Yuang Jiang, and Irene Li. 2023. Large Language Models on WikipediaStyle Survey Generation: an Evaluation in NLP Concepts. arXiv preprint arXiv:2308.10410 (2023). - · arXiv
  20. [25][25] Philip J Guo, Thomas Zimmermann, Nachiappan Nagappan, and Brendan Murphy. 2010. Characterizing and Predicting which Bugs Get Fixed: An Empirical Study of Microsoft Windows. In ACM/IEEE International Conference on Software Engineering (ICSE) . 495-504. https://doi.org/10.1145/1806799.1806871 - · DOI
  21. [26][26] Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't Stop Pretraining: Adapt Language Models to Domains and Tasks. In Annual Meeting of the Association for Computational Linguistics (ACL) . 8342-8360. https://doi.org/10.18653/v1/2020.acl-main.740 - · DOI · Paper page
  22. [27][27] Emitza Guzman, David Azócar, and Yang Li. 2014. Sentiment Analysis of Commit Comments in GitHub: An Empirical Study. In Working Conference on Mining Software Repositories (MSR) . 352-355. https://doi.org/10.1145/2597073.2597118 - · DOI
  23. [28][28] Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari. 2023. Evaluating Large Language Models in Generating Synthetic HCI Research Data: A Case Study. In ACM CHI Conference on Human Factors in Computing Systems (CHI) . 1-19. https://doi.org/10.1145/3544548.3580688 - · DOI · Paper page
  24. [29][29] David Hammer and Leema K Berland. 2014. Confusing Claims for Data: A Critique of Common Practices for Presenting Qualitative Research on Learning. Journal of the Learning Sciences 23, 1 (2014), 37-46. https://doi.org/10.1080/10508406. 2013.802652 - · DOI
  25. [30][30] Tobias Hey, Jan Keim, Anne Koziolek, and Walter F Tichy. 2020. Norbert: Transfer Learning for Requirements Classification. In IEEE International Requirements Engineering Conference (RE) . IEEE, 169-179. https://doi.org/10.1109/ RE48521.2020.00028 -
  26. [31][31] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2023. Large Language Models for Software Engineering: A Systematic Literature Review. arXiv preprint arXiv:2308.10620 (2023). - · arXiv
  27. [32][32] Hennie Huijgens, Ayushi Rastogi, Ernst Mulders, Georgios Gousios, and Arie van Deursen. 2020. Questions for Data Scientists in Software Engineering: A Replication. In ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) . 568-579. https://doi.org/10.1145/3368089. 3409717 - · DOI
  28. [33][33] Brittany Johnson, Rahul Pandita, Justin Smith, Denae Ford, Sarah Elder, Emerson Murphy-Hill, Sarah Heckman, and Caitlin Sadowski. 2016. A Cross-tool Communication Study on Program Analysis Tool nNtifications. In ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE) . 73-84. -
  29. [34][34] Eunice Jun, Maureen Daum, Jared Roesch, Sarah Chasins, Emery Berger, Rene Just, and Katharina Reinecke. 2019. Tea: A High-level Language and Runtime System for Automating Statistical Analysis. In ACM Symposium on User Interface Software and Technology (UIST) . 591-603. https://doi.org/10.1145/3332165.3347940 - · DOI
  30. [35][35] Eunice Jun, Audrey Seo, Jeffrey Heer, and René Just. 2022. Tisane: Authoring Statistical Models via Formal Reasoning from Conceptual and Data Relationships. In ACM CHI Conference on Human Factors in Computing Systems (CHI) . 1-16. https://doi.org/10.1145/3491102.3501888 - · DOI
  31. [36][36] Miryung Kim, Dongxiang Cai, and Sunghun Kim. 2011. An Empirical Investigation into the Role of API-level Refactorings during Software Evolution. In IEEE/ACM International Conference on Software Engineering (ICSE) . 151-160. https://doi.org/10.1145/1985793.1985815 - · DOI
  32. [37][37] Miryung Kim, Thomas Zimmermann, Robert DeLine, and Andrew Begel. 2016. The Emerging Role of Data Scientists on Software Development Teams. In IEEE/ACM International Conference on Software Engineering (ICSE) . 96-107. https://doi.org/10.1109/TSE.2017.2754374 - · DOI · Paper page
  33. [38][38] Miryung Kim, Thomas Zimmermann, Robert DeLine, and Andrew Begel. 2017. Data Scientists in Software Teams: State of the Art and Challenges. IEEE Transactions on Software Engineering 44, 11 (2017), 1024-1038. https://doi.org/10. 1109/TSE.2017.2754374 -
  34. [39][39] Serkan Kirbas, Etienne Windels, Olayori McBello, Kevin Kells, Matthew Pagano, Rafal Szalanski, Vesna Nowack, Emily Rowan Winter, Steve Counsell, David Bowes, et al. 2021. On the Introduction of Automatic Program Repair in Bloomberg. IEEE Software 38, 4 (2021), 43-51. https://doi.org/10.1109/MS.2021.3071086 - · DOI
  35. [40][40] Barbara A. Kitchenham and Shari Lawrence Pfleeger. 2008. Personal Opinion Surveys. In Guide to Advanced Empirical Software Engineering , Forrest Shull, Janice Singer, and Dag I. K. Sjøberg (Eds.). Springer, 63-92. https://doi.org/10. 1007/978-1-84800-044-5\_3 -
  36. [41][41] Amy J Ko, Thomas D LaToza, and Margaret M Burnett. 2015. A Practical Guide to Controlled Experiments of Software Engineering Tools with Human Participants. Empirical Software Engineering 20, 1 (2015), 110-141. https: [//doi.org/10.1007/s10664-013-9279-3](https://doi.org/10.1007/s10664-013-9279-3) - · DOI
  37. [42][42] Jenny T Liang, Maryam Arab, Minhyuk Ko, Amy J Ko, and Thomas D LaToza. 2023. A Qualitative Study on the Implementation Design Decisions of Developers. In IEEE/ACM International Conference on Software Engineering (ICSE) . 435-447. https://doi.org/10.1109/ICSE48619.2023.00047 - · DOI
  38. [45][45] Jenny T Liang, Thomas Zimmermann, and Denae Ford. 2022. Understanding Skills for OSS Communities on GitHub. In ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) . 170-182. https://doi.org/10.1145/3540250.3549082 - · DOI
  39. [47][47] David Lo, Nachiappan Nagappan, and Thomas Zimmermann. 2015. How Practitioners Perceive the Relevance of Software Engineering Research. In ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) . 415-425. https://doi.org/10.1145/2786805.2786809 - · DOI
  40. [48][48] Shantanu Mandal, Adhrik Chethan, Vahid Janfaza, SM Mahmud, Todd A Anderson, Javier Turek, Jesmin Jahan Tithi, and Abdullah Muzahid. 2023. Large Language Models Based Automatic Synthesis of Software Specifications. arXiv preprint arXiv:2304.09181 (2023). - · arXiv
  41. [50][50] OpenAI. 2023. GPT-4 Technical Report. ArXiv abs/2303.08774 (2023). - · arXiv
  42. [51][51] Daniel Pletea, Bogdan Vasilescu, and Alexander Serebrenik. 2014. Security and Emotion: Sentiment Analysis of Security Discussions on GitHub. In Working Conference on Mining Software Repositories (MSR) . 348-351. https: //doi.org/10.1145/2597073.2597117 - · DOI
  43. [52][52] Lutz Prechelt. 2021. On Implicit Assumptions Underlying Software Engineering Research. In International Conference on Evaluation and Assessment in Software Engineering (EASE) . 336-339. https://doi.org/10.1145/3463274.3463356 - · DOI
  44. [53][53] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shotTtext-to-image Generation. In International Conference on Machine Learning (ICML) . PMLR, 8821-8831. -
  45. [54][54] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code LLaMA: Open Foundation Models for Code. arXiv preprint arXiv:2308.12950 (2023). - · arXiv
  46. [57][57] Marija Selakovic and Michael Pradel. 2016. Performance Issues and Optimizations in JavaScript: An Empirical Study. In IEEE/ACM International Conference on Software Engineering (ICSE) . 61-72. https://doi.org/10.1145/2884781.2884829 - · DOI
  47. [58][58] Mikke Tavast, Anton Kunnari, and Perttu Hämäläinen. 2022. Language Models Can Generate Human-like Self-reports of Emotion. In ACM International Conference on Intelligent User Interfaces (IUI) . 69-72. https://doi.org/10.1145/3490100. 3516464 - · DOI
  48. [59][59] Yingchen Tian, Yuxia Zhang, Klaas-Jan Stol, Lin Jiang, and Hui Liu. 2022. What Makes a Good Commit Message?. In IEEE/ACM International Conference on Software Engineering (ICSE) . 2389-2401. https://doi.org/10.1145/3510003.3510205 - · DOI
  49. [60][60] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971 (2023). - · arXiv
  50. [61][61] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. LLaMA 2: Open Foundation and Fine-tuned Chat Models. arXiv preprint arXiv:2307.09288 (2023). - · arXiv
  51. [62][62] David Wadden, Kyle Lo, Lucy Lu Wang, Arman Cohan, Iz Beltagy, and Hannaneh Hajishirzi. 2022. MultiVerS: Improving Scientific Claim Verification with Weak Supervision and Full-document Context. In Findings of the Association for Computational Linguistics: NAACL . 61-76. https://doi.org/10.18653/v1/2022.findings-naacl.6 - · DOI · Paper page
  52. [64][64] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems 35 (2022), 24824-24837. -
  53. [65][65] Moshi Wei, Nima Shiri Harzevili, Yuchao Huang, Junjie Wang, and Song Wang. 2022. Clear: Contrastive Learning for API Recommendation. In IEEE/ACM International Conference on Software Engineering (ICSE) . 376-387. https: //doi.org/10.1145/3510003.3510159 - · DOI
  54. [66][66] Tongshuang Wu, Haiyi Zhu, Maya Albayrak, Alexis Axon, Amanda Bertsch, Wenxing Deng, Ziqi Ding, Bill Guo, Sireesh Gururaja, Tzu-Sheng Kuo, et al. 2023. LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMs. arXiv preprint arXiv:2307.10168 (2023). - · arXiv
  55. [67][67] Ziang Xiao, Xingdi Yuan, Q Vera Liao, Rania Abdelghani, and Pierre-Yves Oudeyer. 2023. Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding. In ACM International Conference on Intelligent User Interfaces (IUI) . 75-78. https://doi.org/10.1145/3581754.3584136 - · DOI
  56. [69][69] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Conference on Empirical Methods in Natural Language Processing (EMNLP) . 3911-3921. https://doi.org/10.18653/v1/D18-1425 - · DOI · Paper page
  57. [70][70] Jialu Zhang, Todd Mytkowicz, Mike Kaufman, Ruzica Piskac, and Shuvendu K Lahiri. 2022. Using Pre-trained Language Models to Resolve Textual and Semantic Merge Conflicts (Experience Paper). In ACM International Symposium on Software Testing and Analysis (ISSTA) . 77-88. https://doi.org/10.1145/3533767.3534396 - · DOI
  58. [71][71] Zibin Zheng, Kaiwen Ning, Jiachi Chen, Yanlin Wang, Wenqing Chen, Lianghong Guo, and Weicheng Wang. 2023. Towards an Understanding of Large Language Models in Software Engineering Tasks. arXiv preprint arXiv:2308.11396 (2023). Received 2023-09-28; accepted 2024-04-16 · arXiv

Cited by

Sorted by citations

Authors
Liang, J. T., Badea, C., Bird, C., DeLine, R., Ford, D., Forsgren, N., & Zimmermann, T.
Affiliations
  • Carnegie Mellon University (US)
  • Microsoft (United States) (US)
ORCID
Venue
Proceedings of the ACM on Software Engineering
Publisher
Association for Computing Machinery (ACM)
Published
2024-07-12
Volume
1
Issue
FSE
Pages
1330-1353
ISSN
2994-970X (ISSN-L)
DOI
10.1145/3660767
OpenAlex
W4400582732
Cited by
19
References
10
Citations by year
2026: 9 · 2025: 9 · 2024: 1
FWCI
7.14
Type
journal article
Language
en
Open access
Licence
Locations
  • Proceedings of the ACM on software engineering. (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (1745016, 2140739)
  • Microsoft Research
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
Keywords
GPT-4, large language models, research assumptions, code generation, software engineering data, production systems, user study, implementation errors

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review