An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation
Open accessThe DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index.
Abstract
Unit tests play a key role in ensuring the correctness of software. However, manually creating unit tests is a laborious task, motivating the need for automation. Large Language Models (LLMs) have recently been applied to various aspects of software development, including their suggested use for automated generation of unit tests, but while requiring additional training or few-shot learning on examples of existing tests. This paper presents a large-scale empirical evaluation on the effectiveness of LLMs for automated unit test generation without requiring additional training or manual effort. Concretely, we consider an approach where the LLM is provided with prompts that include the signature and implementation of a function under test, along with usage examples extracted from documentation. Furthermore, if a generated test fails, our approach attempts to generate a new test that fixes the problem by re-prompting the model with the failing test and error message. We implement our approach inTestPilot, an adaptive LLM-based test generation tool for JavaScript that automatically generates unit tests for the methods in a given project's API. We evaluateTestPilotusing OpenAI'sgpt3.5-turboLLM on 25 npm packages with a total of 1,684 API functions. The generated tests achieve a median statement coverage of 70.2% and branch coverage of 52.8%. In contrast, the state-of-the feedback-directed JavaScript test generation technique, Nessie, achieves only 51.3% statement coverage and 25.6% branch coverage. Furthermore, experiments with excluding parts of the information included in the prompts show that all components contribute towards the generation of effective test suites. We also find that 92.8% ofTestPilot's generated tests have$\leq$50% similarity with existing tests (as measured by normalized edit distance), with none of them being exact copies. Finally, we runTestPilotwith two additional LLMs, OpenAI's oldercode-cushman-002LLM andStarCoder, an LLM for which the training process is publicly documented. Overall, we observed similar results with the former (68.2% median statement coverage), and somewhat worse results with the latter (54.0% median statement coverage), suggesting that the effectiveness of the approach is influenced by the size and training set of the LLM, but does not fundamentally depend on the specific model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Research in this area has explored the use of large language models to generate test cases, showing promising results [19, 47, 108, 140, 178, 232].”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [178]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“Some LLMs such as BERT [30] or GPT-3 [31] are trained purely on text extracted from books and other public sources, while others like OpenAI Codex [32] and AlphaCode [33] are put through additional training on publicly available source code to make them better suited for software development tasks…”
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“Some LLMs such as BERT [30] or GPT-3 [31] are trained purely on text extracted from books and other public sources, while others like OpenAI Codex [32] and AlphaCode [33] are put through additional training on publicly available source code to make them better suited for software development tasks…”
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“Some LLMs such as BERT [30] or GPT-3 [31] are trained purely on text extracted from books and other public sources, while others like OpenAI Codex [32] and AlphaCode [33] are put through additional training on publicly available source code to make them better suited for software development tasks…”
BackgroundKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · cited 2×
“Similarly, Tufano et al.’s A THENA T EST [26] generates tests using a BART transformer model [44] fine-tuned on a training set of functions and their corresponding tests.”
Extreme Programming Explained: Embrace Change
Beck, K. L., & Andres, C. (2000) · Cited by 5,902
CUTE: a concolic unit testing engine for C
Sen, K., Marinov, D., & Agha, G. (2005) · Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1081706.1081750 · Cited by 1,651 · cited 2×
“This fact has inspired extensive research on techniques for automated test generation including fuzzing [5], [6], feedback-directed random test generation [7], [8], [9], [10], [11], dynamic symbolic execution [12], [13], [14], [15], and search-based and evolutionary techniques [16], [17].”
BackgroundMethodDART: directed automated random testing
Godefroid, P., Klarlund, N., & Sen, K. (2005) · Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation · doi:10.1145/1065010.1065036 · Cited by 1,570 · cited 2×
“This fact has inspired extensive research on techniques for automated test generation including fuzzing [5], [6], feedback-directed random test generation [7], [8], [9], [10], [11], dynamic symbolic execution [12], [13], [14], [15], and search-based and evolutionary techniques [16], [17].”
BackgroundMethodDominance statistics: Ordinal analyses to answer ordinal questions.
Cliff, N. (1993) · Psychological Bulletin · doi:10.1037//0033-2909.114.3.494 · Cited by 1,485 · cited 2×
“Both these differences are statistically significant (p-values 0.002 and 0.027 respectively) with a large effect size, measured by Cliff’s delta [55], of 0.493 for statement coverage and a medium one (0.431) for branch coverage.”
BackgroundMethodInfluentialDefects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447 · cited 1×
“In experiments on 5 projects from Defects4J [69], AthenaTest generated 158K test cases, achieving similar test coverage as EvoSuite [16], a popular search-based test generation tool, and covering 43% of all focal methods.”
MethodEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 1×
“Some LLMs such as BERT [30] or GPT-3 [31] are trained purely on text extracted from books and other public sources, while others like OpenAI Codex [32] and AlphaCode [33] are put through additional training on publicly available source code to make them better suited for software development tasks…”
BackgroundRegression testing minimization, selection and prioritization: a survey
Yoo, S., & Harman, M. (2012) · Software Testing, Verification and Reliability · doi:10.1002/stv.430 · Cited by 1,322
An empirical study of the reliability of UNIX utilities
Miller, B. P., Fredriksen, L., & So, B. (1990) · Communications of the ACM · doi:10.1145/96267.96279 · Cited by 1,281
EvoSuite: automatic test suite generation for object-oriented software
Fraser, G., & Arcuri, A. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025179 · Cited by 1,110 · cited 4×
“This fact has inspired extensive research on techniques for automated test generation including fuzzing [5], [6], feedback-directed random test generation [7], [8], [9], [10], [11], dynamic symbolic execution [12], [13], [14], [15], and search-based and evolutionary techniques [16], [17].”
InfluentialSchleimer, S., S, W. D., & Aiken, A. (2003) · Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data · doi:10.1145/872757.872770 · Cited by 1,099
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 1×
“…BERT [30] or GPT-3 [31] are trained purely on text extracted from books and other public sources, while others like OpenAI Codex [32] and AlphaCode [33] are put through additional training on publicly available source code to make them better suited for software development tasks [34], [35], [36],…”
BackgroundCompetition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952 · cited 1×
“Following Reynolds and McDonell [47], we posit that providing the model with input-output examples or performing additional training is not necessary and that careful prompt crafting is sufficient.”
Learning to Represent Programs with GraphsPreprint
Allamanis, M., Brockschmidt, M., & Khademi, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00740 · Cited by 845 · cited 2×
“Neural techniques are rapidly being adopted for solving various Software Engineering problems, with promising results in several domains including code completion [34], [35], [36], [37], [38], program repair [39], [40], [41], and bug-finding [42], [43].”
MethodEXE: automatically generating inputs of death
Cadar, C., Ganesh, V., Pawlowski, P. M., Dill, D. L., & Engler, D. R. (2006) · Proceedings of the 13th ACM conference on Computer and communications security · doi:10.1145/1180405.1180445 · Cited by 788 · cited 2×
“This fact has inspired extensive research on techniques for automated test generation including fuzzing [5], [6], feedback-directed random test generation [7], [8], [9], [10], [11], dynamic symbolic execution [12], [13], [14], [15], and search-based and evolutionary techniques [16], [17].”
BackgroundMethodEvaluating large language models trained on code
Chen, M. (2024) · TIB Data Manager · doi:10.57702/wbv3e61b · Cited by 737
Randoop: feedback-directed random testing for Java
Pacheco, C., & Ernst, M. D. (2007) · Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion · doi:10.1145/1297846.1297902 · Cited by 558 · cited 6×
“They position their technique as a complement to traditional techniques [8], [17].”
BackgroundMethodInfluentialIntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474 · cited 2×
“Neural techniques are rapidly being adopted for solving various Software Engineering problems, with promising results in several domains including code completion [34], [35], [36], [37], [38], program repair [39], [40], [41], and bug-finding [42], [43].”
MethodA Symbolic Execution Framework for JavaScript
Saxena, P., Akhawe, D., Hanna, S., Mao, F., McCamant, S., & Song, D. (2010) · 2010 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2010.38 · Cited by 455 · cited 1×
“Saxena et al. [73] present Kudzu, a tool that aims to find injection vulnerabilities in client-side JavaScript applications by exploring an application’s input space.”
BackgroundA fast bit-vector algorithm for approximate string matching based on dynamic programming
Myers, G. (1999) · Journal of the ACM · doi:10.1145/316542.316550 · Cited by 445 · cited 1×
“Instead, they find that measuring similarity through edit distance [60] produces more meaningful results.”
BackgroundJCrasher: an automatic robustness tester for Java
Csallner, C., & Smaragdakis, Y. (2004) · Software: Practice and Experience · doi:10.1002/spe.602 · Cited by 426 · cited 3×
“This fact has inspired extensive research on techniques for automated test generation including fuzzing [5], [6], feedback-directed random test generation [7], [8], [9], [10], [11], dynamic symbolic execution [12], [13], [14], [15], and search-based and evolutionary techniques [16], [17].”
BackgroundMethodDeepFix: Fixing Common C Language Errors by Deep Learning
Gupta, R., Pal, S., Kanade, A., & Shevade, S. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10742 · Cited by 426 · cited 2×
“Neural techniques are rapidly being adopted for solving various Software Engineering problems, with promising results in several domains including code completion [34], [35], [36], [37], [38], program repair [39], [40], [41], and bug-finding [42], [43].”
MethodPanichella, A., Kifetew, F. M., & Tonella, P. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2663435 · Cited by 352 · cited 5×
“Their results show that, on most of their target 27 Python projects, their proposed technique, C ODA M OSA , outperforms the baseline search-based technique, Pynguin’s implementation of M OSA [64], as well as using only Codex.”
BackgroundMethodResultInfluentialUnit Test Case Generation with Transformers and Focal ContextPreprint
Tufano, M., Drain, D., Svyatkovskiy, A., Deng, S. K., & Sundaresan, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.05617 · Cited by 346 · cited 4×
“Similarly, Tufano et al.’s A THENA T EST [26] generates tests using a BART transformer model [44] fine-tuned on a training set of functions and their corresponding tests.”
InfluentialDeepBugs: a learning approach to name-based bug detection
Pradel, M., & Sen, K. (2018) · Proceedings of the ACM on Programming Languages · doi:10.1145/3276517 · Cited by 342 · cited 2×
“Neural techniques are rapidly being adopted for solving various Software Engineering problems, with promising results in several domains including code completion [34], [35], [36], [37], [38], program repair [39], [40], [41], and bug-finding [42], [43].”
MethodProceedings of the 44th International Conference on Software Engineering
(2022) · doi:10.1145/3510003 · Cited by 326
Search-Based Software Testing: Past, Present and Future
McMinn, P. (2011) · 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops · doi:10.1109/icstw.2011.100 · Cited by 322 · cited 1×
“…an alternative to (and potentially complements) traditional techniques for automated test generation, including feedback-directed random test generation [7], [8], [9], [10], [11], search-based and evolutionary techniques [16], [17], [63], [64], and dynamic symbolic execution [12], [13], [14], [15].”
BackgroundCodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models
Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00085 · Cited by 306 · cited 6×
“Recently, Lemieux et al. [27] reported that code Fig.”
BackgroundMethodInfluentialRegression testing minimization, selection and prioritization: a survey
Yoo, S., & Harman, M. (2012) · Software Testing, Verification and Reliability · doi:10.1002/stvr.430 · Cited by 297
Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks
Mastropaolo, A., Scalabrino, S., Cooper, N., Nader Palacio, D., Poshyvanyk, D., Oliveto, R., & Bavota, G. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00041 · Cited by 247 · cited 7×
“We are aware of several recent research efforts in which LLMs are used for test generation [23], [24], [25], [26], [27], [28], [29].”
BackgroundMethodInfluentialCode Prediction by Feeding Trees to Transformers
Kim, S., Zhao, J., Tian, Y., & Chandra, S. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00026 · Cited by 209 · cited 2×
“Neural techniques are rapidly being adopted for solving various Software Engineering problems, with promising results in several domains including code completion [34], [35], [36], [37], [38], program repair [39], [40], [41], and bug-finding [42], [43].”
MethodA framework for automated testing of javascript web applications
Artzi, S., Dolby, J., Jensen, S. H., Møller, A., & Tip, F. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985871 · Cited by 199 · cited 1×
“Artemis [74] is a framework for automated test generation that iteratively generates tests for client-side JavaScript applications consisting of sequences of events, using a heuristics-based strategy that considers the locations read and written by each event handler to focus on the generation of…”
MethodSelf-citationBeck, K. (1999) · Proceedings Technology of Object-Oriented Languages and Systems. TOOLS 29 (Cat. No.PR00275) · doi:10.1109/tools.1999.779100 · Cited by 196
Big code != big vocabulary: open-vocabulary models for source code
Karampatsis, R.-M., Babii, H., Robbes, R., Sutton, C., & Janes, A. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380342 · Cited by 182 · cited 2×
“Neural techniques are rapidly being adopted for solving various Software Engineering problems, with promising results in several domains including code completion [34], [35], [36], [37], [38], program repair [39], [40], [41], and bug-finding [42], [43].”
MethodA Survey on Unit Testing Practices and Problems
Daka, E., & Fraser, G. (2014) · 2014 IEEE 25th International Symposium on Software Reliability Engineering · doi:10.1109/issre.2014.11 · Cited by 175 · cited 1×
“However, creating unit tests by hand is labor-intensive and tedious, causing some developers to skip writing tests altogether [4].”
BackgroundEvolutionary Generation of Whole Test Suites
Fraser, G., & Arcuri, A. (2011) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2011.19 · Cited by 170 · cited 4×
“This fact has inspired extensive research on techniques for automated test generation including fuzzing [5], [6], feedback-directed random test generation [7], [8], [9], [10], [11], dynamic symbolic execution [12], [13], [14], [15], and search-based and evolutionary techniques [16], [17].”
BackgroundMethodInfluentialAn Industrial Evaluation of Unit Test Generation: Finding Real Faults in a Financial Application
Almasi, M. M., Hemmati, H., Fraser, G., Arcuri, A., & Benefelds, J. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP) · doi:10.1109/icse-seip.2017.27 · Cited by 163 · cited 1×
“First, the generated tests are typically less readable and understandable than manually written tests [18], [19], especially due to the use of unintuitive variable names [20].”
BackgroundModeling readability to improve unit tests
Daka, E., Campos, J., Fraser, G., Dorn, J., & Weimer, W. (2015) · Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2786805.2786838 · Cited by 143 · cited 1×
“First, the generated tests are typically less readable and understandable than manually written tests [18], [19], especially due to the use of unintuitive variable names [20].”
BackgroundSelf-Supervised Bug Detection and RepairPreprint
Allamanis, M., Jackson-Flux, H., & Brockschmidt, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.12787 · Cited by 141 · cited 2×
“Neural techniques are rapidly being adopted for solving various Software Engineering problems, with promising results in several domains including code completion [34], [35], [36], [37], [38], program repair [39], [40], [41], and bug-finding [42], [43].”
Global Relational Models of Source Code
Hellendoorn, V. J., Sutton, C. A., Singh, R., & Maniatis, P. (2020) · International Conference on Learning Representations · Cited by 130
On the diffusion of test smells in automatically generated test code: an empirical study
Palomba, F., Nucci, D. D., Panichella, A., Oliveto, R., & Lucia, A. D. (2016) · IEEE/ACM International Workshop on Search-Based Software Testing (SBST) · doi:10.1145/2897010.2897016 · Cited by 129 · cited 2×
“Similarly, given that it is often difficult for traditional test generation techniques to generate (useful) assertions [21], [22], A TLAS [28] uses LLMs to generate an assert statement for a given (assertion-less) Java test.”
BackgroundMethodTOGA: a neural method for test oracle generation
Dinella, E., Ryan, G., Mytkowicz, T., & Lahiri, S. K. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510141 · Cited by 118 · cited 1×
“TOGA [67] uses similar techniques but additionally incorporates an exceptional oracle classifier to decide if a given method requires an assertion to test exceptional behavior.”
MethodCode Completion with Neural Attention and Pointer Networks
Li, J., Wang, Y., Lyu, M. R., & King, I. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/578 · Cited by 112 · cited 2×
“Neural techniques are rapidly being adopted for solving various Software Engineering problems, with promising results in several domains including code completion [34], [35], [36], [37], [38], program repair [39], [40], [41], and bug-finding [42], [43].”
MethodOn learning meaningful assert statements for unit test cases
Watson, C., Tufano, M., Moran, K., Bavota, G., & Poshyvanyk, D. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380429 · Cited by 111 · cited 7×
“We are aware of several recent research efforts in which LLMs are used for test generation [23], [24], [25], [26], [27], [28], [29].”
BackgroundMethodInfluentialFinding errors in .net with feedback-directed random testing
Pacheco, C., Lahiri, S. K., & Ball, T. (2008) · International Symposium on Software Testing and Analysis · doi:10.1145/1390630.1390643 · Cited by 102 · cited 4×
“This fact has inspired extensive research on techniques for automated test generation including fuzzing [5], [6], feedback-directed random test generation [7], [8], [9], [10], [11], dynamic symbolic execution [12], [13], [14], [15], and search-based and evolutionary techniques [16], [17].”
BackgroundMethodInfluentialSymJS: automatic symbolic testing of JavaScript web applications
Li, G., Andreasen, E., & Ghosh, I. (2014) · ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE) · doi:10.1145/2635868.2635913 · Cited by 100 · cited 1×
“Li et al. [75] extends Artemis with dynamic symbolic execution to improve its ability to explore the value space, and Tanida et al. [76] further improve on this work by augmenting generated test inputs with user-supplied invariants.”
Generating accurate assert statements for unit test cases using pretrained transformers
Tufano, M., Drain, D., Svyatkovskiy, A., & Sundaresan, N. (2022) · Proceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test · doi:10.1145/3524481.3527220 · Cited by 99 · cited 4×
“…goal differences with ATLAS [28] and Mastrapaolo et al.’s [29], [45] work above, both these efforts also require a data set of test methods (with assertions) and their corresponding focal methods, whether to use in the main training [28] or in fine tuning during transfer learning [29], [45], [46].”
BackgroundInfluentialAssessing Oracle Quality with Checked Coverage
Schüler, D., & Zeller, A. (2011) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2011.32 · Cited by 97 · cited 1×
“Measuring assertion/checked coverage as defined by Schuler and Zeller [62] is a possible alternative, but this is practically difficult to implement precisely for JavaScript.”
MethodAn empirical investigation on the readability of manual and generated test cases
Grano, G., Scalabrino, S., Gall, H. C., & Oliveto, R. (2018) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3196321.3196363 · Cited by 93 · cited 1×
“First, the generated tests are typically less readable and understandable than manually written tests [18], [19], especially due to the use of unintuitive variable names [20].”
BackgroundA Survey of Dynamic Analysis and Test Generation for JavaScript
Andreasen, E., Gong, L., Møller, A., Pradel, M., Selakovic, M., Sen, K., & Staicu, C.-A. (2017) · ACM Computing Surveys · doi:10.1145/3106739 · Cited by 92 · cited 1×
“Severalprevious projects have considered test generation for JavaScript (see [72] for a survey).”
BackgroundCodeT: Code Generation with Generated Tests
Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J.-G., & Chen, W. (2022) · arXiv · doi:10.48550/arxiv.2207.10397 · Cited by 92 · cited 3×
“Differing Goals: T I C ODER [24] and C ODE T [23] use Codex to generate implementations and test cases from problem descriptions expressed in natural language.”
Leveraging existing tests in automated test generation for web applications
Fard, A. M., Mirzaaghaei, M., & Mesbah, A. (2014) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2642937.2642991 · Cited by 82
Using Transfer Learning for Code-Related Tasks
Mastropaolo, A., Cooper, N., Palacio, D. N., Scalabrino, S., Poshyvanyk, D., Oliveto, R., & Bavota, G. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3183297 · Cited by 72 · cited 5×
“In fact, in addition to the goal differences with ATLAS [28] and Mastrapaolo et al.’s [29], [45] work above, both these efforts also require a data set of test methods (with assertions) and their corresponding focal methods, whether to use in the main training [28] or in fine tuning during transfer…”
BackgroundMethodInfluentialAutomated assertion generation via information retrieval and its integration with deep learning
Yu, H., Lou, Y., Sun, K., Ran, D., Xie, T., Hao, D., Li, Y., Li, G., & Wang, Q. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510149 · Cited by 63 · cited 1×
“With the same goal, Mastrapaolo et al. [29], [45] and Tufano et al. [46] perform follow up work using transfer learning, while Yu et al. [66] use information retrieval techniques to further improve the assert statements generated by Atlas.”
Using GitHub Copilot for Test Generation in Python: An Empirical Study
Haji, K. E., Brandt, C., & Zaidman, A. (2024) · doi:10.1145/3644032.3644443 · Cited by 56
Revisiting Test Smells in Automatically Generated Tests: Limitations, Pitfalls, and Opportunities
Panichella, A., Panichella, S., Fraser, G., Sawant, A. A., & Hellendoorn, V. J. (2020) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme46990.2020.00056 · Cited by 55 · cited 2×
“Similarly, given that it is often difficult for traditional test generation techniques to generate (useful) assertions [21], [22], A TLAS [28] uses LLMs to generate an assert statement for a given (assertion-less) Java test.”
BackgroundMethodTufano, M., Deng, S. K., Sundaresan, N., & Svyatkovskiy, A. (2022) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3524842.3528009 · Cited by 51
JSEFT: Automated Javascript Unit Test Generation
Mirshokraie, S., Mesbah, A., & Pattabiraman, K. (2015) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2015.7102595 · Cited by 47 · cited 2×
“Mirshokraie et al. [81], [82] present an approach in which tests are generated for client-side JavaScript applications consisting of sequences of events.”
Using search-based algorithms for Ajax event sequence generation during testing
Marchetto, A., & Tonella, P. (2010) · Empirical Software Engineering · doi:10.1007/s10664-010-9149-1 · Cited by 41 · cited 1×
“Marchetto and Tonella [78] present a search-based test generation technique that constructs tests consisting of sequences of events that relies on the automatic extraction of a finite state machine that represents that application’s state.”
BackgroundPradel, M., & Chandra, S. (2021) · Communications of the ACM · doi:10.1145/3460348 · Cited by 41 · cited 1×
“Pradel and Chandra [65] survey the current state of the art in this emerging research area.”
BackgroundJSART: JavaScript Assertion-Based Regression Testing
Mirshokraie, S., & Mesbah, A. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-31753-8_18 · Cited by 40 · cited 2×
“Several test generation tools for JavaScript are capable of generating tests containing assertions.”
BackgroundMethodTest generation for higher-order functions in dynamic languages
Selakovic, M., Pradel, M., Karim, R., & Tip, F. (2018) · Proceedings of the ACM on Programming Languages · doi:10.1145/3276531 · Cited by 40 · cited 4×
“This fact has inspired extensive research on techniques for automated test generation including fuzzing [5], [6], feedback-directed random test generation [7], [8], [9], [10], [11], dynamic symbolic execution [12], [13], [14], [15], and search-based and evolutionary techniques [16], [17].”
InfluentialSelf-citationTowards More Realistic Evaluation for Neural Test Oracle Generation
Liu, Z., Liu, K., Xia, X., & Yang, X. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598080 · Cited by 40 · cited 1×
“While TOGA can be integrated with EvoSuite [16] to create an end-to-end test-generation tool, recent work [68] points out several shortcomings of the evaluation methods, casting doubt on the validity of the reported results.”
MethodTransferring an automated test generation tool to practice: from pex to fakes and code digger
Tillmann, N., de Halleux, J., & Xie, T. (2014) · Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/2642937.2642941 · Cited by 38 · cited 2×
“This fact has inspired extensive research on techniques for automated test generation including fuzzing [5], [6], feedback-directed random test generation [7], [8], [9], [10], [11], dynamic symbolic execution [12], [13], [14], [15], and search-based and evolutionary techniques [16], [17].”
BackgroundMethodType Regression Testing to Detect Breaking Changes in Node.js Libraries
Mezzetti, G., Møller, A., & Torp, M. T. (2018) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.ecoop.2018.7 · Cited by 37 · cited 1×
“Our notion of an access path takes a somewhat simplified form compared to the original concept proposed by Mezzetti et al. [51], and consists of a package name followed by a sequence of property names.”
Nessie: automatically testing JavaScript APIs with asynchronous callbacks
Arteca, E., Harner, S., Pradel, M., & Tip, F. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510106 · Cited by 34 · cited 10×
“External Validity: Despite our evaluation scale signifi-cantly exceeding evaluations of previous test generation approaches [11], [25], our results are still based on 25 npm packages and may not generalize to other JavaScript code bases.”
BackgroundMethodResultInfluentialSelf-citationCode Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Code
Bareiß, P., Souza, B., d’Amorim, M., & Pradel, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.01335 · Cited by 27 · cited 6×
“External Validity: Despite our evaluation scale signifi-cantly exceeding evaluations of previous test generation approaches [11], [25], our results are still based on 25 npm packages and may not generalize to other JavaScript code bases.”
BackgroundMethodInfluentialInteractive Code Generation via Test-Driven User-Intent Formalization
Lahiri, S. K., Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., Musuvathi, M., Choudhury, P., von Veh, C., Inala, J. P., Wang, C., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.05950 · Cited by 26 · cited 3×
“Differing Goals: T I C ODER [24] and C ODE T [23] use Codex to generate implementations and test cases from problem descriptions expressed in natural language.”
BackgroundMethodPYTHIA: Generating test cases with oracles for JavaScript applications
Mirshokraie, S., Mesbah, A., & Pattabiraman, K. (2013) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2013.6693121 · Cited by 22 · cited 2×
“Mirshokraie et al. [81], [82] present an approach in which tests are generated for client-side JavaScript applications consisting of sequences of events.”
BackgroundGenerating Fixtures for JavaScript Unit Testing (T)
Fard, A. M., Mesbah, A., & Wohlstadter, E. (2015) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.26 · Cited by 18 · cited 1×
“Fard et al. [77] present ConFix, a tool that uses a combination of dynamic analysis and symbolic execution to automatically generate instances of the Document Object Model (DOM) that can serve as test fixtures in unit tests for client-side JavaScript code.”
MethodGuess What: Test Case Generation for Javascript with Unsupervised Probabilistic Type Inference
Stallenberg, D., Olsthoorn, M., & Panichella, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21251-2_5 · Cited by 9 · cited 1×
“Other Techniques: Stallenberg et al. [70] present a test generation technique for JavaScript based on unsupervised type inference consisting of three phases.”
MethodAutomatic Unit Test Generation and Execution for JavaScript Program through Symbolic Execution
Hideo, T., Li, G., Ghosh, I., & Uehara, T. (2014) · Cited by 4
Introduction: Experience Reports
Shore, J. (n.d.) · AGILE 2006 (AGILE'06) · doi:10.1109/agile.2006.36
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · arXiv · doi:10.48550/arxiv.2102.07350
Research works printed but not linked: 2
- Siddiqui, 2021, Learning Test-Driven Development
- 10.1145/3276531 · DOI
Cited by
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029
Demystifying LLM-Based Software Engineering Agents
Xia, C. S., Deng, Y., Dunn, S., & Zhang, L. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3715754 · Cited by 849
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
“For example, in their work for unit test case generation, Alagarsamy et al. [29] additionally introduce a verification method to check and repair the naming consistency (i.e., revising the test method name to be consistent with the focal method name) and the test signatures (i.e., adding missing…”
Self-collaboration code generation via ChatGPT
Dong, Y., Jiang, X., Jin, Z., & Li, G. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3672459 · Cited by 256
“(2) This setting is mainly suited for function-level code generation and proves challenging to extend to file-level or repository-level code generation.”
Self-collaboration Code Generation via ChatGPT
Dong, Y., Jiang, X., Jin, Z., & Li, G. (2023) · arXiv · doi:10.48550/arxiv.2304.07590 · Cited by 256
Metadata
- Authors
- Schäfer, M., Nadi, S., Eghbali, A., & Tip, F.
- Affiliations
- GitHub, Kidlington, U.K.
- University of Alberta (CA)
- University of Stuttgart (DE)
- Northeastern University (US)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2024-01
- Volume
- 50
- Issue
- 1
- Pages
- 85-105
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W4389104713
- Cited by
- 431
- References
- 70
- Citations by year
- 2026: 164 · 2025: 187 · 2024: 78
- FWCI
- 299.17
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (CCF-1907727, CCF-2307742)
- Canada Research Chairs
- Natural Sciences and Engineering Research Council of Canada (RGPIN-2017-04289)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- automated unit test generation, large language models, prompt engineering, JavaScript, LLM model comparison, StarCoder, npm packages
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.