CoChat
Sign Up
PreprintCited by 14Software Engineering ResearchSoftware Engineering Techniques and PracticesScientific Computing and Data Management

The DOI matches its DataCite record.

How Scientists Use Large Language Models to Program

Open access
O'Brien, Gabrielle·2025·arXiv
DOI 10.48550/arxiv.2502.17348arXiv 2502.17348
Open this paper in CoChat — free account

Abstract

Scientists across disciplines write code for critical activities like data collection and generation, statistical modeling, and visualization. As large language models that can generate code have become widely available, scientists may increasingly use these models during research software development. We investigate the characteristics of scientists who are early-adopters of code generating models and conduct interviews with scientists at a public, research-focused university. Through interviews and reviews of user interaction logs, we see that scientists often use code generating models as an information retrieval tool for navigating unfamiliar programming languages and libraries. We present findings about their verification strategies and discuss potential vulnerabilities that may emerge from code generation practices unknowingly influencing the parameters of scientific analyses.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.17348

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Still being checked.

Research works printed but not linked: 101

  1. [1][1] 2016. Retraction: Inferring the effects of cancer treatment: Divergent results from Early Breast Cancer Trialists' Collaborative Group meta-analyses of randomized trials and observational data from SEER registries (Journal of Clinical Oncology (34) 803-809 (2016)). Journal of Clinical Oncology 34, 27 (9 2016), 3358-3359. https://doi.org/10.1200/JCO.2016.69.0875/ASSET/IMAGES/LARGE/ JCO690875TA5.JPEG - · DOI
  2. [2][2] Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel. 2024. Can LLMs Replace Manual Annotation of Software Engineering Artifacts? (8 2024). https://arxiv.org/abs/2408.05534v1 - · arXiv
  3. [3][3] Sarah Dahlby Albright, Titus H. Klinge, and Samuel A. Rebelsky. 2018. A functional approach to data science in CS1. In SIGCSE 2018 - Proceedings of the 49th ACM Technical Symposium on Computer Science Education , Vol. 2018January. https://doi.org/10.1145/3159450.3159550 - · DOI
  4. [5][5] Shraddha Barke, Michael B. James, and Nadia Polikarpova. 2023. Grounded Copilot: How Programmers Interact with Code-Generating Models. Proceedings of the ACM on Programming Languages 7, OOPSLA1 (4 2023). https://doi.org/ 10.1145/3586030 - · DOI · Paper page
  5. [6][6] Victor R. Basili, Jeffrey C. Carver, Daniela Cruzes, Lorin M. Hochstein, Jeffrey K. Hollingsworth, Forrest Shull, and Marvin V. Zelkowitz. 2008. Understanding the high-performance-computing community: A software engineer's perspective. IEEE Software 25, 4 (7 2008), 29-36. https://doi.org/10.1109/MS.2008.103 - · DOI
  6. [8][8] Susan M Baxter, Steven W Day, Jacquelyn S Fetrow, and Stephanie J Reisinger. 2006. Scientific Software Development Is Not an Oxymoron PLoS Computational Biology | www. 2 (2006), 975. https://doi.org/10.1371/journal.pcbi.0020087 - · DOI
  7. [9][9] Brett A. Becker, Paul Denny, James Finnie-Ansley, Andrew Luxton-Reilly, James Prather, and Eddie Antonio Santos. 2023. Programming Is Hard - or at Least It Used to Be: Educational Opportunities and Challenges of AI Code Generation. SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education 1 (3 2023), 500-506. https://doi.org/10.1145/3545945.3569759 - · DOI · Paper page
  8. [10][10] Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big?. In FAccT 2021 - Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency . https://doi.org/10.1145/3442188.3445922 - · DOI · Paper page
  9. [11][11] Shreya Bhatia, Tarushi Gandhi, Dhruv Kumar, and Pankaj Jalote. 2024. Unit Test Generation using Generative AI : A Comparative Performance Analysis of Autogeneration Tools. (4 2024), 54-61. https://doi.org/10.1145/3643795.3648396 - · DOI · Paper page
  10. [12][12] Leo Breiman. 2001. Statistical modeling: The two cultures. Statist. Sci. 16, 3 (2001), 199-215. https://doi.org/10.1214/SS/1009213726 - · DOI · Paper page
  11. [13][13] Jeffrey Carver, Dustin Heaton, Lorin Hochstein, and Roscoe Bartlett. 2013. Selfperceptions about software engineering: A survey of scientists and engineers. Computing in Science and Engineering 15, 1 (2013), 7-11. https://doi.org/10. 1109/MCSE.2013.12 -
  12. [14][14] Jeffrey C. Carver, Nic Weber, Karthik Ram, Sandra Gesing, and Daniel S. Katz. 2022. A survey of the state of the practice for research software in the United - States. PeerJ Computer Science 8 (5 2022), e963. https://doi.org/10.7717/PEERJCS.963 - · DOI
  13. [15][15] Shubham Chandel, Colin B. Clement, Guillermo Serrato, and Neel Sundaresan. 2022. Training and Evaluating a Jupyter Notebook Data Science Assistant. (1 2022). https://arxiv.org/abs/2201.12901v1 - · arXiv
  14. [16][16] Sarah Chasins, Elena Glassman, Joshua Sunshine, Ningzhi Tang, Meng Chen, Zheng Ning, Aakash Bansal, Yu Huang, Collin Mcmillan, Toby Jia, and Jun Li. 2023. An Empirical Study of Developer Behaviors for Validating and Repairing AI-Generated Code. (3 2023). https://doi.org/10.1184/R1/22223533.V1 - · DOI
  15. [17][17] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. (7 2021). http://arxiv.org/abs/2107.03374 - · arXiv
  16. [18][18] Ruijia Cheng, Ruotong Wang, Thomas Zimmermann, and Denae Ford. 2024. 'It would work for me too': How online communities shape software developers' trust in AI-powered code generation tools. ACM Transactions on Interactive Intelligent Systems 14, 2 (5 2024). https://doi.org/10.1145/3651990/ASSET/6FD6E872-408C-4446-9A0B60EC7D004732/ASSETS/GRAPHIC/TIIS-2023-JUN-004-F05.JPG - · DOI
  17. [19][19] Juliet Corbin and Anselm Strauss. 2012. Basics of Qualitative Research (3rd ed.): Techniques and Procedures for Developing Grounded Theory . https://doi.org/10. 4135/9781452230153 -
  18. [20][20] Zheyuan (Kevin) Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz. 2024. The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers. (9 2024). https: //doi.org/10.2139/SSRN.4945566 - · DOI · Paper page
  19. [21][21] Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, and Zhen Ming (Jack) Jiang. 2023. GitHub Copilot AI pair programmer: Asset or Liability? Journal of Systems and Software 203 (9 2023), 111734. https://doi.org/10.1016/J.JSS.2023.111734 - · DOI · Paper page
  20. [22][22] Victor Dibia. 2023. LIDA: A Tool for Automatic Generation of GrammarAgnostic Visualizations and Infographics using Large Language Models. Proceedings of the Annual Meeting of the Association for Computational Linguistics 3 (3 2023), 113-126. https://doi.org/10.18653/v1/2023.acl-demo.11 - · DOI
  21. [23][23] David Donoho. 2017. 50 Years of Data Science. https://doi.org/10.1080/10618600. 2017.1384734 - · DOI
  22. [24][24] Pierre Dragicevic, Yvonne Jansen, Abhraneel Sarma, Matthew Kay, and Fanny Chevalier. 2019. Increasing the transparency of research papers with explorable multiverse analyses. Conference on Human Factors in Computing Systems Proceedings 15 (5 2019), 978-979. https://doi.org/10.1145/3290605.3300295 - · DOI
  23. [25][25] Shubhang Shekhar Dvivedi, Vyshnav Vijay, Sai Leela Rahul Pujari, Shoumik Lodh, and Dhruv Kumar. 2024. A Comparative Analysis of Large Language Models for Code Documentation Generation. AIware 2024 - Proceedings of the 1st ACMInternational Conference on AI-Powered Software, Co-located with: ESEC/FSE 2024 (7 2024), 65-73. https://doi.org/10.1145/3664646.3664765 - · DOI
  24. [26][26] Christof Ebert and Panos Louridas. 2023. Generative AI for Software Practitioners. IEEE Software 40, 4 (2023). https://doi.org/10.1109/MS.2023.3265877 - · DOI · Paper page
  25. [27][27] Molly Q. Feldman and Carolyn Jane Anderson. 2024. Non-Expert Programmers in the Generative AI Future. ACM International Conference Proceeding Series (6 2024). https://doi.org/10.1145/3663384.3663393 - · DOI
  26. [29][29] Ken Gu, Eunice Jun, and Tim Althoff. 2023. Understanding and Supporting Debugging Workflows in Multiverse Analysis. Conference on Human Factors in Computing Systems - Proceedings (4 2023). https://doi.org/10.1145/3544548. 3581099 - · DOI · Paper page
  27. [30][30] Ken Gu, Ruoxi Shang, Tim Althoff, Chenglong Wang, and Steven M Drucker. 2024. How Do Analysts Understand and Verify AI-Assisted Data Analyses? Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '24), May 11â·fi16, 2024, Honolulu, HI, USA 1, 24 (2024). https://doi.org/10.1145/ 3613904.3642497 -
  28. [31][31] Jo Erskine Hannay, Carolyn MacLeod, Janice Singer, Hans Petter Langtangen, Dietmar Pfahl, and Greg Wilson. 2009. How do scientists develop and use scientific software? Proceedings of the 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering, SECSE 2009 (2009), 1-8. https://doi.org/10.1109/SECSE.2009.5069155 - · DOI
  29. [32][32] Dustin Heaton and Jeffrey C. Carver. 2015. Claims about the use of software engineering practices in science: A systematic literature review. Information and Software Technology 67 (11 2015), 207-219. https://doi.org/10.1016/J.INFSOF. 2015.07.011 - · DOI
  30. [34][34] Tony Hey, Stewart Tansley, Kristin Michele Tolle, and others. 2009. The fourth paradigm: data-intensive scientific discovery . Vol. 1. Microsoft research Redmond, WA. -
  31. [35][35] Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. 2022. Instruction Induction: From Few Examples to Natural Language Task Descriptions. Proceedings of the Annual Meeting of the Association for Computational Linguistics 1 (5 2022), 1935-1952. https://doi.org/10.18653/v1/2023.acl-long.108 - · DOI
  32. [36][36] James Howison, Ewa Deelman, Michael J. McLennan, Rafael Ferreira Da Silva, and James D. Herbsleb. 2015. Understanding the scientific software ecosystem and its impact: Current and future measures. Research Evaluation 24, 4 (10 2015), 454-470. https://doi.org/10.1093/RESEVAL/RVV014 - · DOI
  33. [37][37] James Howison and James D Herbsleb. 2010. Scientific software production and collaboration. CSCW. Anais (2010). -
  34. [38][38] James Howison and James D. Herbsleb. 2011. Scientific software production: Incentives and collaboration. Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW (2011), 513-522. https://doi.org/10.1145/ 1958824.1958904 -
  35. [39][39] Saki Imai. 2022. Is GitHub Copilot a Substitute for Human Pair-programming? An Empirical Study. Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings (7 2022), 319-321. https: //doi.org/10.1145/3510454 - · DOI
  36. [40][40] Dhanya Jayagopal, Justin Lubin, and Sarah E. Chasins. 2022. Exploring the Learnability of Program Synthesizers by Novice Programmers. UIST 2022 Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (10 2022). https://doi.org/10.1145/3526113.3545659 - · DOI · Paper page
  37. [41][41] Arne N. Johanson and Wilhelm Hasselbring. 2018. Software Engineering for Computational Science. Computing in Science and Engineering 20, 2 (3 2018), 90-109. https://doi.org/10.1109/MCSE.2018.021651343 - · DOI
  38. [42][42] Lucas N. Joppa, Greg McInerny, Richard Harper, Lara Salido, Kenji Takeda, Kenton O'Hara, David Gavaghan, and Stephen Emmott. 2013. Troubling trends in scientific software use. Science 340, 6134 (5 2013), 814-815. https://doi.org/ 10.1126/SCIENCE.1231535/SUPPL{\_}FILE/814.MP3 - · DOI
  39. [43][43] Gregor Jošt, Viktor Taneski, and Sašo Karakatič. 2024. The Impact of Large Language Models on Programming Education and Student Learning Outcomes. Applied Sciences 2024, Vol. 14, Page 4115 14, 10 (5 2024), 4115. https://doi.org/10. 3390/APP14104115 -
  40. [44][44] Samia Kabir, David N. Udo-Imeh, Bonan Kou, and Tianyi Zhang. 2024. Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions. Conference on Human Factors in Computing Systems - Proceedings (5 2024). https://doi.org/10.1145/3613904.3642596 - · DOI · Paper page
  41. [45][45] Sean Kandel, Andreas Paepcke, Joseph M. Hellerstein, and Jeffrey Heer. 2012. Enterprise data analysis and visualization: An interview study. IEEE Transactions on Visualization and Computer Graphics 18, 12 (2012). https://doi.org/10.1109/ TVCG.2012.219 -
  42. [46][46] Upulee Kanewala and James M. Bieman. 2014. Testing scientific software: A systematic literature review. Information and Software Technology 56, 10 (2014), 1219-1232. https://doi.org/10.1016/J.INFSOF.2014.05.006 - · DOI · Paper page
  43. [47][47] Amelia Karraker and Kenzie Latham. 2015. Authors' explanation of the retraction. Journal of Health and Social Behavior 56, 3 (1 2015), 417-419. https://doi.org/10.1177/0022146515595817 - · DOI
  44. [48][48] Diane Kelly. 2015. Scientific software development viewed as knowledge acquisition: Towards understanding the development of risk-averse scientific software. Journal of Systems and Software 109 (11 2015), 50-61. https: //doi.org/10.1016/J.JSS.2015.07.027 - · DOI
  45. [49][49] Diane F. Kelly. 2007. A software chasm: Software engineering and scientific computing. IEEE Software 24, 6 (11 2007). https://doi.org/10.1109/MS.2007.155 - · DOI
  46. [50][50] Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2018. A systematic literature review of automated feedback generation for programming exercises. ACM Transactions on Computing Education 19, 1 (9 2018). https: //doi.org/10.1145/3231711/SUPPL{\_}FILE/KEUNING.ZIP - · DOI
  47. [51][51] Sean Kross and Philip J. Guo. 2019. Practitioners Teaching Data Science in Industry and Academia. https://doi.org/10.1145/3290605.3300493 - · DOI
  48. [52][52] Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-Tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. DS1000: A Natural and Reliable Benchmark for Data Science Code Generation. Asian Journal of Clinical Pediatrics and Neonatology 2, 3 (7 2023), 18319-18345. https://proceedings.mlr.press/v202/lai23b.html -
  49. [53][53] Michael J. Lee and Andrew J. Ko. 2011. Personifying programming tool feedback improves novice programmers' learning. ICER'11 - Proceedings of the ACM SIGCSE 2011 International Computing Education Research Workshop (2011), 109116. https://doi.org/10.1145/2016911.2016934 - · DOI · Paper page
  50. [54][54] Annie Li, Madeline Endres, and Westley Weimer. 2022. Debugging with stack overflow: Web search behavior in novice and expert programmers. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Software Engineering Education and Training . 69-81. -
  51. [55][55] Jenny T. Liang, Chenyang Yang, and Brad A. Myers. 2024. A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges. Proceedings - International Conference on Software Engineering (2 2024). https: //doi.org/10.1145/3597503.3608128 - · DOI · Paper page
  52. [56][56] Jiali Liu, Nadia Boukhelifa, and James R. Eagan. 2020. Understanding the Role of Alternatives in Data Analysis Practices. IEEE Transactions on Visualization and Computer Graphics 26, 1 (1 2020), 66-76. https://doi.org/10.1109/TVCG. 2019.2934593 - · DOI
  53. [57][57] Michael Xieyang Liu, Advait Sarkar, Carina Negreanu, Benjamin Zorn, Jack Williams, Neil Toronto, and Andrew D. Gordon. 2023. "What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models. Conference on Human Factors in Computing Systems -Proceedings (4 2023). https://doi.org/10.1145/3544548.3580817/ASSET/F8C36C77-80E5-4C61-B5F140A6BB94ECDE/ASSETS/HTML/IMAGES/CHI23-163-IMG70.SVG - · DOI
  54. [58][58] Rongxin Liu, Carter Zenke, Charlie Liu, Andrew Holmes, Patrick Thornton, and David J. Malan. 2024. Teaching CS50 with AI: Leveraging Generative Artificial Intelligence in Computer Science Education. SIGCSE 2024 - Proceedings of the 55th ACM Technical Symposium on Computer Science Education 1 (3 2024), 750-756. https://doi.org/10.1145/3626252.3630938 - · DOI · Paper page
  55. [59][59] Francesca Lucchetti, Zixuan Wu, Arjun Guha, Molly Q Feldman, and Carolyn Jane Anderson. 2024. Substance Beats Style: Why Beginning Students Fail to Code with LLMs. (10 2024). https://arxiv.org/abs/2410.19792v1 - · arXiv
  56. [60][60] Piush J. Mandhane. 2024. Notice of Retraction: Hahn LM, et al. Post-COVID19 Condition in Children. JAMA Pediatrics. 2023;177(11):1226-1228. JAMA Pediatrics (2024). https://doi.org/10.1001/JAMAPEDIATRICS.2024.2996 - · DOI
  57. [61][61] Lauren E. Margulieux, James Prather, Brent N. Reeves, Brett A. Becker, Gozde Cetin Uzun, Dastyni Loksa, Juho Leinonen, and Paul Denny. 2024. SelfRegulation, Self-Efficacy, and Fear of Failure Interactions with How Novices Use LLMs to Solve Programming Problems. Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE 1 (7 2024), 276-282. https://doi.org/10.1145/3649217.3653621 - · DOI
  58. [63][63] Andrew M. McNutt, Chenglong Wang, Robert A. Deline, and Steven M. Drucker. 2023. On the Design of AI-powered Code Assistants for Notebooks. Conference on Human Factors in Computing Systems - Proceedings 16, 23 (4 2023). https: //doi.org/10.1145/3544548.3580940 - · DOI
  59. [64][64] Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2024. GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models. (10 2024). https: //arxiv.org/abs/2410.05229v1 -
  60. [65][65] Asaf Achi Mordechai, Yoav Goldberg, and Reut Tsarfaty. 2024. NoviCode: Generating Programs from Natural Language Utterances by Novices. (7 2024). https://arxiv.org/abs/2407.10626v2 - · arXiv
  61. [66][66] Hussein Mozannar, Adam Fourney, Gagan Bansal, and Eric Horvitz. 2024. Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming. Conference on Human Factors in Computing Systems - Proceedings (5 2024). https://doi.org/10.1145/3613904.3641936 - · DOI · Paper page
  62. [67][67] Marcus R. Munafò, Brian A. Nosek, Dorothy V.M. Bishop, Katherine S. Button, Christopher D. Chambers, Nathalie Percie Du Sert, Uri Simonsohn, Eric Jan Wagenmakers, Jennifer J. Ware, and John P.A. Ioannidis. 2017. A manifesto for reproducible science. Nature Human Behaviour 1, 1 (1 2017). https://doi.org/10. 1038/S41562-016-0021 -
  63. [68][68] Daye Nam, Andrew MacVean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an LLM to Help with Code Understanding. Proceedings - International Conference on Software Engineering (2024), 1184-1196. https: //doi.org/10.1145/3597503.3639187 - · DOI · Paper page
  64. [69][69] Udit Nangia, Daniel S Katz, and others. 2017. Track 1 paper: surveying the US National Postdoctoral Association regarding software use and training in research. In Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE 5.1) . -
  65. [70][70] National Academies of Sciences and Policy and Global Affairs and Board on Research Data and Information and Division on Engineering and Physical Sciences and Committee on Applied and Theoretical Statistics and Board on Mathematical Sciences. 2019. Reproducibility and replicability in science . National Academie Press. -
  66. [71][71] Sydney Nguyen, Hannah Mclean Babe, Yangtian Zi, Arjun Guha, Carolyn Jane Anderson, Molly Q Feldman, and Jane Anderson. 2024. How Beginning Programmers and Code LLMs (Mis)read Each Other. (2024), 26. https://doi.org/10. 1145/3613904.3642706 -
  67. [72][72] Brian A Nosek, Tom E Hardwicke, Hannah Moshontz, Aurélien Allard, Katherine S Corker, Anna Dreber, Fiona Fidler, Joe Hilgard, Melissa Kline Struhl, Michèle B Nuijten, Julia M Rohrer, Felipe Romero, Anne M Scheel, Laura D 20. Scherer, Felix D Schönbrodt, and Simine Vazire. 2024. Replicability, Robustness, and Reproducibility in Psychological Science. 52 (2024). https: //doi.org/10.1146/annurev-psych-020821 - · DOI
  68. [73][73] Jeremy E. Oakley and Anthony O'Hagan. 2004. Probabilistic Sensitivity Analysis of Complex Models: A Bayesian Approach. Journal of the Royal Statistical Society Series B: Statistical Methodology 66, 3 (8 2004), 751-769. https://doi.org/10.1111/ J.1467-9868.2004.05304.X -
  69. [75][75] Gabrielle O'Brien and Jordan Mick. 2024. In the Academy, Data Science Is Lonely: Barriers to Adopting Data Science Methods for Scientific Research. Harvard Data Science Review 6, 2 (2 2024). https://doi.org/10.1162/99608F92.7CA04767 - · DOI
  70. [76][76] Aadarsh Padiyath, Xinying Hou, Amy Pang, Diego Viramontes Vargas, Xingjian Gu, Tamara Nelson-Fromm, Zihan Wu, Mark Guzdial, and Barbara Ericson. 2024. Insights from Social Shaping Theory: The Appropriation of Large Language Models in an Undergraduate Programming Course. Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 (8 2024), 114-130. https://doi.org/10.1145/3632620.3671098 - · DOI · Paper page
  71. [77][77] Pruthvi Patel, Swaroop Mishra, Mihir Parmar, and Chitta Baral. 2022. Is a Question Decomposition Unit All We Need? Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022 (5 2022), 4553-4569. https://doi.org/10.18653/v1/2022.emnlp-main.302 - · DOI
  72. [78][78] Prakash Prabhu, Hanjun Kim, Taewook Oh, Thomas B. Jablin, Nick P. Johnson, Matthew Zoufaly, Arun Raman, Feng Liu, David Walker, Yun Zhang, Soumyadeep Ghosh, David I. August, Jialu Huang, and Stephen Beard. 2011. A survey of the practice of computational science. State of the Practice Reports, SC'11 (2011). https://doi.org/10.1145/2063348.2063374 - · DOI
  73. [79][79] James Prather, Brent N Reeves, Paul Denny, Brett A Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett Powell, James Finnie-Ansley, Gar-Rett Powell, Eddie Antonio Santos, ; P Denny, J Leinonen, A Luxton-Reilly, J Finnie-Ansley, B A Becker, and E Antonio Santos. 2023. 'It's Weird That it Knows What I Want': Usability and Interactions with Copilot for Novice Programmers. ACM Transactions on Computer-Human Interaction 31, 4 (2023). https://doi.org/10. 1145/3617367 -
  74. [80][80] James Prather, Brent N Reeves, Juho Leinonen, Stephen MacNeil, Arisoa S Randrianasolo, Brett A. Becker, Bailey Kimmel, Jared Wright, and Ben Briggs. 2024. The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers. Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 (8 2024), 469-486. https: //doi.org/10.1145/3632620.3671116 - · DOI · Paper page
  75. [81][81] Sruti Srinivasa Ragavan, Zhitao Hou, Yun Wang, Andrew D. Gordon, Haidong Zhang, and Dongmei Zhang. 2022. GridBook: Natural Language Formulas for the Spreadsheet Grid. International Conference on Intelligent User Interfaces, Proceedings IUI (3 2022), 345-368. https://doi.org/10.1145/3490099.3511161/ SUPPL{\_}FILE/IUI22-59-VIDEO-540P.MP4 - · DOI
  76. [82][82] G Rothermel, M Shaw, and S Wiedenbeck. 2011. The state of the art in end-user software engineering. ACM Comput. Surv 43 (2011), 44. https://doi.org/10.1145/ 1922649.1922658 -
  77. [83][83] Peter J. Rousseeuw. 1987. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20, C (11 1987), 53-65. https://doi.org/10.1016/0377-0427(87)90125-7 - · DOI · Paper page
  78. [84][84] Daniel Russo. 2023. Navigating the Complexity of Generative AI Adoption in Software Engineering. ACM Transactions on Software Engineering and Methodology 37, 111 (7 2023), 44. https://doi.org/10.1145/1122445.1122456 - · DOI
  79. [86][86] Advait Sarkar, Andrew D. Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022. What is it like to program with artificial intelligence? (8 2022). https://arxiv.org/abs/2208.06213v2 - · arXiv
  80. [87][87] Abhraneel Sarma, Alex Kale, Michael Jongho Moon, Nathan Taback, Fanny Chevalier, Jessica Hullman, and Matthew Kay. 2023. multiverse: Multiplexing Alternative Data Analyses in R Notebooks. Conference on Human Factors in Computing Systems - Proceedings (4 2023). https://doi.org/10.1145/3544548. 3580726 - · DOI · Paper page
  81. [88][88] Max Schafer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. IEEE Transactions on Software Engineering 50, 1 (1 2024), 85-105. https://doi. org/10.1109/TSE.2023.3334955 - · DOI · Paper page
  82. [89][89] Judith Segal. 2007. Some problems of professional end user developers. In IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2007) . 111-118. -
  83. [90][90] Judith Segal. 2009. Some challenges facing software engineers developing software for scientists. Proceedings of the 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering, SECSE 2009 (2009), 9-14. https://doi.org/10.1109/SECSE.2009.5069156 - · DOI
  84. [91][91] Benjamin Hayden Sims. 2021. Research software engineering: Professionalization, roles, and identity . Technical Report. Los Alamos National Laboratory (LANL), Los Alamos, NM (United States). https://doi.org/10.2172/1845242 - · DOI
  85. [92][92] Spencer Smith, Malavika Srinivasan, and Sumanth Shankar. 2019. Debunking the myth that upfront requirements are infeasible for scientific computing software. Proceedings - 2019 IEEE/ACM 14th International Workshop on Software Engineering for Science, SE4Science 2019 (5 2019), 33-40. https://doi.org/10.1109/ SE4SCIENCE.2019.00011 -
  86. [93][93] Sara Steegen, Francis Tuerlinckx, Andrew Gelman, and Wolf Vanpaemel. 2016. Increasing Transparency Through a Multiverse Analysis. Perspectives on Psychological Science 11, 5 (9 2016), 702-712. https://doi.org/10.1177/1745691616658637 - · DOI
  87. [94][94] Tim Storer. 2017. Bridging the chasm: A survey of sofware engineering practice in scientific programming. Comput. Surveys 50, 4 (8 2017). https://doi.org/10. 1145/3084225 -
  88. [95][95] Lev Tankelevitch, Viktor Kewenig, Auste Simkute, Ava Elizabeth, Scott Advait Sarkar Abigail Sellen, Sean Rintel, Ava Elizabeth Scott, Advait Sarkar, and Abigail Sellen. [n. d.]. The Metacognitive Demands and Opportunities of Generative AI. ([n. d.]). https://doi.org/10.1145/3613904.3642902 - · DOI · Paper page
  89. [96][96] Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. Conference on Human Factors in Computing Systems Proceedings (4 2022). https://doi.org/10.1145/3491101.3519665 - · DOI · Paper page
  90. [97][97] Zhijie Wang, Zijie Zhou, Da Song, Yuheng Huang, Shengmai Chen, Lei Ma, and Tianyi Zhang. 2024. Where Do Large Language Models Fail When Generating Code? (6 2024). https://arxiv.org/abs/2406.08731v2 - · arXiv
  91. [98][98] Thomas Weber, Maximilian Brandmaier, Albrecht Schmidt, and Sven Mayer. 2024. Significant Productivity Gains through Programming with Large Language Models. Proceedings of the ACM on Human-Computer Interaction 8, EICS (6 2024). https://doi.org/10.1145/3661145 - · DOI · Paper page
  92. [99][99] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems 35, 3 (12 2022), 24824-24837. https://aijournals.com/index. php/ajcpn/article/view/184 -
  93. [101][101] Justin D. Weisz, Michael Muller, Stephanie Houde, John Richards, Steven I. Ross, Fernando Martinez, Mayank Agarwal, and Kartik Talamadupula. 2021. Perfection Not Required? Human-AI Partnerships in Code Translation. International Conference on Intelligent User Interfaces, Proceedings IUI (4 2021), 402-412. https://doi.org/10.1145/3397481.3450656 - · DOI · Paper page
  94. [102][102] Justin D. Weisz, Michael Muller, Steven I. Ross, Fernando Martinez, Stephanie Houde, Mayank Agarwal, Kartik Talamadupula, and John T. Richards. 2022. Better Together? An Evaluation of AI-Supported Code Translation. International Conference on Intelligent User Interfaces, Proceedings IUI (3 2022), 369-391. https: //doi.org/10.1145/3490099.3511157 - · DOI · Paper page
  95. [103][103] Igor Wiese, Ivanilton Polato, and Gustavo Pinto. 2020. Naming the Pain in Developing Scientific Software. IEEE Software 37, 4 (7 2020), 75-82. https: //doi.org/10.1109/MS.2019.2899838 - · DOI
  96. [104][104] Greg Wilson. 2016. Software Carpentry: Lessons learned. https://doi.org/10. 12688/f1000research.3-62.v2 -
  97. [105][105] Greg Wilson, D. A. Aruliah, C. Titus Brown, Neil P. Chue Hong, Matt Davis, Richard T. Guy, Steven H. D. Haddock, Katy Huff, Ian M. Mitchell, Mark Plumbley, Ben Waugh, Ethan P. White, and Paul Wilson. 2012. Best Practices for Scientific Computing. Science 340, 6134 (9 2012), 814-815. https: //doi.org/10.1371/journal.pbio.1001745 - · DOI · Paper page
  98. [106][106] Frank F. Xu, Bogdan Vasilescu, and Graham Neubig. 2022. In-IDE Code Generation from Natural Language: Promise and Challenges. ACM Transactions on Software Engineering and Methodology 31, 2 (4 2022). https://doi.org/10.1145/ 3487569 -
  99. [107][107] Burak Yetistiren, Isik Ozsoy, and Eray Tuzun. 2022. Assessing the quality of GitHub copilot's code generation. PROMISE 2022 - Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering, co-located with ESEC/FSE 2022 10, 22 (11 2022), 62-71. https: //doi.org/10.1145/3558489.3559072 - · DOI
  100. [108][108] Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, and others. 2022. Natural language to code generation in interactive data science notebooks. arXiv preprint arXiv:2212.09248 (2022). - · arXiv
  101. [109][109] Albert Ziegler, Eirini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022. Productivity assessment of neural code completion. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming . 21-29.

Cited by

Sorted by citations

See all 14

Metadata

Authors
O'Brien, G.
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
1-16
DOI
10.48550/arxiv.2502.17348
arXiv
2502.17348
OpenAlex
W4409766894
Cited by
14
References
0
Citations by year
2026: 12 · 2025: 2
FWCI
16.76
Type
conference paper
Language
en
Open access
Licence
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
  • Deep Blue (University of Michigan) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
Alfred P. Sloan Foundation (g-2024-22425)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
Keywords
large language models, programming, code generation, natural language programming, AI-assisted software development

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: How Scientists Use Large Language Models to Program. https://app.cochat.ai/papers/10.48550/arxiv.2502.17348
The paper
O'Brien, G. (2025). How Scientists Use Large Language Models to Program. arXiv. https://doi.org/10.48550/arxiv.2502.17348