The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Can generative AI bridge the gap? A quasi-experimental study of non-programmers with AI vs. programmers without AI
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Statistical power analyses using G∗Power 3.1: tests for correlation and regression analyses
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.4.1149 · Cited by 37,615
Experimentation in Software Engineering
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012) · doi:10.1007/978-3-642-29044-2 · Cited by 4,460
Requirements engineering: a roadmap
Nuseibeh, B., & Easterbrook, S. (2000) · Proceedings of the Conference on The Future of Software Engineering · doi:10.1145/336512.336523 · Cited by 1,830
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Four dark corners of requirements engineering
Zave, P., & Jackson, M. (1997) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/237432.237434 · Cited by 734
The state of the art in end-user software engineering
Ko, A. J., Abraham, R., Beckwith, L., Blackwell, A., Burnett, M., Erwig, M., Scaffidi, C., Lawrance, J., Lieberman, H., Myers, B., et al. (2011) · ACM Computing Surveys · doi:10.1145/1922649.1922658 · Cited by 633
Cliff´s Delta Calculator: A non-parametric effect size program for two groups of observations
Macbeth, G., Razumiejczyk, E., & Ledesma, R. D. (2010) · Universitas Psychologica · doi:10.11144/javeriana.upsy10-2.cdcp · Cited by 442
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580919 · Cited by 366
Reporting Experiments in Software Engineering
Jedlitschka, A., Ciolkowski, M., & Pfahl, D. (2008) · Guide to Advanced Empirical Software Engineering · doi:10.1007/978-1-84800-044-5_8 · Cited by 331
Computing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316 · cited 2×
“…(e.g., Barke et al. 2023 ; Ziegler et al. 2024; Sheikhaei et al. 2024), on novices and stu - dents using code LLMs in educational settings (e.g., Denny et al. 2024; Prather et al. 2023 ; Kazemitabaar et al. 2023 ), and on non-programmers and end users interacting with AI to specify or generate…”
Towards an understanding of large language models in software engineering tasks
Zheng, Z., Ning, K., Zhong, Q., Chen, J., Chen, W., Guo, L., Wang, W., & Wang, Y. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10602-0 · Cited by 283
How Beginning Programmers and Code LLMs (Mis)read Each Other
Nguyen, S., Babe, H. M., Zi, Y., Guha, A., Anderson, C. J., & Feldman, M. Q. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642706 · Cited by 160
How secure is AI-generated code: a large-scale comparison of large language models
Tihanyi, N., Bisztray, T., Ferrag, M. A., Jain, R., & Cordeiro, L. C. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10590-1 · Cited by 132
Measuring GitHub Copilot's Impact on Productivity
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2024) · Communications of the ACM · doi:10.1145/3633453 · Cited by 124 · cited 6×
“…ecological validity and feasibility: (i) a group of non-programmers without AI would likely exhibit a predictably failing performance, pro - viding only a redundant baseline, and (ii) a group of programmers with AI has already been extensively studied with consistent findings (Ziegler et al. 2024).”
InfluentialExplainable automated debugging via large language model-driven scientific debugging
Kang, S., Chen, B., Yoo, S., & Lou, J. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10594-x · Cited by 119
AI-Assisted Code Authoring at Scale: Fine-Tuning, Deploying, and Mixed Methods Evaluation
Murali, V., Maddila, C., Ahmad, I., Bolin, M., Cheng, D., Ghorbani, N., Fernandez, R., Nagappan, N., & Rigby, P. C. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643774 · Cited by 114
Using AI-based coding assistants in practice: State of affairs, perceptions, and ways forward
Sergeyuk, A., Golubev, Y., Bryksin, T., & Ahmed, I. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2024.107610 · Cited by 101
Do you Really Code? Designing and Evaluating Screening Questions for Online Surveys with Programmers
Danilova, A., Naiakshina, A., Horstmann, S. A., & Smith, M. (2021) · International Conference on Software Engineering · doi:10.1109/icse43902.2021.00057 · Cited by 80
Ajimati, M. O., Carroll, N., & Maher, M. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112300 · Cited by 63
Hao, H., Hasan, K. A., Qin, H., Macedo, M., Tian, Y., Ding, S. H. H., & Hassan, A. E. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10540-x · Cited by 51
Non-Expert Programmers in the Generative AI Future
Feldman, M. Q., & Anderson, C. J. (2024) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3663384.3663393 · Cited by 50
Sheikhaei, M. S., Tian, Y., Wang, S., & Xu, B. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10548-3 · Cited by 45
Exploring the Learnability of Program Synthesizers by Novice Programmers
Jayagopal, D., Lubin, J., & Chasins, S. E. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545659 · Cited by 40
AI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluation
Murali, V., Maddila, C., Ahmad, I., Bolin, M., Cheng, D., Ghorbani, N., Fernandez, R., Nagappan, N., & Rigby, P. C. (2023) · arXiv · doi:10.48550/arxiv.2305.12050 · Cited by 33
Shadish, W. R. (2001) · International Encyclopedia of the Social & Behavioral Sciences · doi:10.1016/b0-08-043076-7/00419-8 · Cited by 24
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · cited 5×
“On the other hand, several authors contend that rather than replacing programmers, AI will transform their roles and competencies (Barke et al. 2023 ; Hao et al. 2024).”
InfluentialPickering, M. A., Williams, H., Gan, A., He, W., Park, H., Vélez, F., Littman, M. L., & Ur, B. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713271 · Cited by 21
Ma, Q., Koedinger, K., & Wu, T. (2025) · arXiv · doi:10.48550/arxiv.2509.21890
Research works printed but not linked: 2
- Blanco-Llano J, Rodríguez-Hernández AG (2011) Revisión, verificación y validación en un proceso de desarrollo de software. Ingeniería Ind 32(1):28–36
- Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Lawrence Erlbaum Associates, Hillsdale, NJ
Metadata
- Authors
- Esnaola, L. M., Ramón, H. D., & Lanzarini, L. C.
- Affiliations
- National University of Northwestern Buenos Aires (AR)
- Universidad Nacional de La Plata (AR)
- Corresponding author
- Esnaola, Leonardo Martín
- Ramón, Hugo Dionisio
- Lanzarini, Laura Cristina
- Venue
- Empirical Software Engineering
- Published
- 2026-02-12
- Volume
- 31
- Issue
- 3
- ISSN
- 1382-3256 (ISSN-L)
- 1573-7616
- OpenAlex
- W7128743205
- Semantic Scholar
- ebe3d1fa9b0abcf8fc0c5b6251dddea505e0b616
- Cited by
- 2
- References
- 26
- Citations by year
- 2026: 2
- FWCI
- 17.01
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Empirical Software Engineering (publishedVersion)
- Funders
- Universidad Nacional del Noroeste de la Provincia de Buenos Aires (Resolución CS 2647/2025 – EXP 5611/2024)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- generative AI, AI-assisted programming, non-programmers, conversational programming, ChatGPT, code generation, task completion time
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.